IDEAS home Printed from https://ideas.repec.org/a/bla/jorssc/v71y2022i4p987-1013.html
   My bibliography  Save this article

Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles

Author

Listed:
  • Jie Min
  • Yili Hong
  • Caleb B. King
  • William Q. Meeker

Abstract

Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modelling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modelling and analysing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new non‐parametric model based on monotonic splines to describe the event process and to estimate the cumulative baseline intensity function of the event process. We develop inference procedures for selecting the best models, quantifying uncertainty and testing heterogeneity in the event process. We then analyse the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems. This paper has online supplementary materials.

Suggested Citation

  • Jie Min & Yili Hong & Caleb B. King & William Q. Meeker, 2022. "Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(4), pages 987-1013, August.
    • Handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013
    DOI: 10.1111/rssc.12564
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssc.12564
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssc.12564?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Simon N. Wood & Natalya Pya & Benjamin Säfken, 2016. "Smoothing Parameter and Model Selection for General Smooth Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1548-1563, October.
    2. Li Xu & Chris Gotwalt & Yili Hong & Caleb B. King & William Q. Meeker, 2020. "Applications of the Fractional-Random-Weight Bootstrap," The American Statistician, Taylor & Francis Journals, vol. 74(4), pages 345-358, October.
    3. Kevin Burke & M. C. Jones & Angela Noufaily, 2020. "A flexible parametric modelling framework for survival analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(2), pages 429-457, April.
    4. Vinayak V Dixit & Sai Chand & Divya J Nair, 2016. "Autonomous Vehicles: Disengagements, Accidents and Reaction Times," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-14, December.
    5. Hong, Yili & Escobar, Luis A. & Meeker, William Q., 2010. "Coverage probabilities of simultaneous confidence bands and regions for log-location-scale distributions," Statistics & Probability Letters, Elsevier, vol. 80(7-8), pages 733-738, April.
    6. J. F. Lawless, 1995. "The Analysis of Recurrent Events for Multiple Subjects," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 44(4), pages 487-498, December.
    7. Kalra, Nidhi & Paddock, Susan M., 2016. "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?," Transportation Research Part A: Policy and Practice, Elsevier, vol. 94(C), pages 182-193.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cian Ryan & Finbarr Murphy & Martin Mullins, 2019. "Semiautonomous Vehicle Risk Analysis: A Telematics‐Based Anomaly Detection Approach," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 1125-1140, May.
    2. Kassens-Noor, Eva & Dake, Dana & Decaminada, Travis & Kotval-K, Zeenat & Qu, Teresa & Wilson, Mark & Pentland, Brian, 2020. "Sociomobility of the 21st century: Autonomous vehicles, planning, and the future city," Transport Policy, Elsevier, vol. 99(C), pages 329-335.
    3. Mohamed Abdel-Aty & Shengxuan Ding, 2024. "A matched case-control analysis of autonomous vs human-driven vehicle accidents," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    4. Ryan, Cian & Murphy, Finbarr & Mullins, Martin, 2020. "Spatial risk modelling of behavioural hotspots: Risk-aware path planning for autonomous vehicles," Transportation Research Part A: Policy and Practice, Elsevier, vol. 134(C), pages 152-163.
    5. Andrea Bertolini & Massimo Riccaboni, 2021. "Grounding the case for a European approach to the regulation of automated driving: the technology-selection effect of liability rules," European Journal of Law and Economics, Springer, vol. 51(2), pages 243-284, April.
    6. Adam Braima S. Mastor & Abdulaziz S. Alghamdi & Oscar Ngesa & Joseph Mung’atu & Christophe Chesneau & Ahmed Z. Afify, 2023. "The Extended Exponential-Weibull Accelerated Failure Time Model with Application to Sudan COVID-19 Data," Mathematics, MDPI, vol. 11(2), pages 1-26, January.
    7. Alatawneh, Anas & Torok, Adam, 2025. "Projecting AV sales in the EU-27 and UK: Insights from Euro emission standards and historical trends," Transport Policy, Elsevier, vol. 163(C), pages 91-101.
    8. Henry X. Liu & Shuo Feng, 2024. "Curse of rarity for autonomous vehicles," Nature Communications, Nature, vol. 15(1), pages 1-5, December.
    9. Zoltan Ferenc Magosi & Christoph Wellershaus & Viktor Roland Tihanyi & Patrick Luley & Arno Eichberger, 2022. "Evaluation Methodology for Physical Radar Perception Sensor Models Based on On-Road Measurements for the Testing and Validation of Automated Driving," Energies, MDPI, vol. 15(7), pages 1-20, March.
    10. Hudson, John & Orviska, Marta & Hunady, Jan, 2019. "People’s attitudes to autonomous vehicles," Transportation Research Part A: Policy and Practice, Elsevier, vol. 121(C), pages 164-176.
    11. Robert F. Balshaw & C. B. Dean, 2002. "A Semiparametric Model for the Analysis of Recurrent-Event Panel Data," Biometrics, The International Biometric Society, vol. 58(2), pages 324-331, June.
    12. Sun, Tianyu & Chand, Satish & Sharpe, Keiran, 2018. "Effect of aging on housing prices: evidence from a panel data," MPRA Paper 94418, University Library of Munich, Germany, revised 01 Mar 2019.
    13. Liu, Peng & Zhang, Yawen & He, Zhen, 2019. "The effect of population age on the acceptable safety of self-driving vehicles," Reliability Engineering and System Safety, Elsevier, vol. 185(C), pages 341-347.
    14. Haiqing Hu & Pandu R. Tadikamalla, 2020. "When to launch a sales promotion for online fashion products? An empirical study," Electronic Commerce Research, Springer, vol. 20(4), pages 737-756, December.
    15. Stefano Cabras & J. James Reade & J.D. Tena, 2018. "Social Pressure or Rational Reactions to Incentives? A Historical Analysis of Reasons for Referee Bias in the Spanish Football," Working Papers 20189, University of Liverpool, Department of Economics.
    16. Anja M. Hahn & Konstantin A. Kholodilin & Sofie R. Waltl & Marco Fongoni, 2024. "Forward to the Past: Short-Term Effects of the Rent Freeze in Berlin," Management Science, INFORMS, vol. 70(3), pages 1901-1923, March.
    17. Hazel Si Min Lim & Araz Taeihagh, 2019. "Algorithmic Decision-Making in AVs: Understanding Ethical and Technical Concerns for Smart Cities," Sustainability, MDPI, vol. 11(20), pages 1-28, October.
    18. Michele Rocca & Marcella Niglio & Marialuisa Restaino, 2024. "Bootstrapping binary GEV regressions for imbalanced datasets," Computational Statistics, Springer, vol. 39(1), pages 181-213, February.
    19. Peter Pütz & Thomas Kneib, 2018. "A penalized spline estimator for fixed effects panel data models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 102(2), pages 145-166, April.
    20. Ursula Laa & Dianne Cook, 2020. "Using tours to visually investigate properties of new projection pursuit indexes with application to problems in physics," Computational Statistics, Springer, vol. 35(3), pages 1171-1205, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.