IDEAS home Printed from https://ideas.repec.org/a/bla/jorssc/v71y2022i4p987-1013.html
   My bibliography  Save this article

Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles

Author

Listed:
  • Jie Min
  • Yili Hong
  • Caleb B. King
  • William Q. Meeker

Abstract

Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modelling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modelling and analysing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new non‐parametric model based on monotonic splines to describe the event process and to estimate the cumulative baseline intensity function of the event process. We develop inference procedures for selecting the best models, quantifying uncertainty and testing heterogeneity in the event process. We then analyse the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems. This paper has online supplementary materials.

Suggested Citation

  • Jie Min & Yili Hong & Caleb B. King & William Q. Meeker, 2022. "Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(4), pages 987-1013, August.
    • Handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013
    DOI: 10.1111/rssc.12564
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssc.12564
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssc.12564?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Vinayak V Dixit & Sai Chand & Divya J Nair, 2016. "Autonomous Vehicles: Disengagements, Accidents and Reaction Times," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-14, December.
    2. Hong, Yili & Escobar, Luis A. & Meeker, William Q., 2010. "Coverage probabilities of simultaneous confidence bands and regions for log-location-scale distributions," Statistics & Probability Letters, Elsevier, vol. 80(7-8), pages 733-738, April.
    3. J. F. Lawless, 1995. "The Analysis of Recurrent Events for Multiple Subjects," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 44(4), pages 487-498, December.
    4. Kevin Burke & M. C. Jones & Angela Noufaily, 2020. "A flexible parametric modelling framework for survival analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(2), pages 429-457, April.
    5. Kalra, Nidhi & Paddock, Susan M., 2016. "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?," Transportation Research Part A: Policy and Practice, Elsevier, vol. 94(C), pages 182-193.
    6. Simon N. Wood & Natalya Pya & Benjamin Säfken, 2016. "Smoothing Parameter and Model Selection for General Smooth Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1548-1563, October.
    7. Li Xu & Chris Gotwalt & Yili Hong & Caleb B. King & William Q. Meeker, 2020. "Applications of the Fractional-Random-Weight Bootstrap," The American Statistician, Taylor & Francis Journals, vol. 74(4), pages 345-358, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cian Ryan & Finbarr Murphy & Martin Mullins, 2019. "Semiautonomous Vehicle Risk Analysis: A Telematics‐Based Anomaly Detection Approach," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 1125-1140, May.
    2. Kassens-Noor, Eva & Dake, Dana & Decaminada, Travis & Kotval-K, Zeenat & Qu, Teresa & Wilson, Mark & Pentland, Brian, 2020. "Sociomobility of the 21st century: Autonomous vehicles, planning, and the future city," Transport Policy, Elsevier, vol. 99(C), pages 329-335.
    3. Ryan, Cian & Murphy, Finbarr & Mullins, Martin, 2020. "Spatial risk modelling of behavioural hotspots: Risk-aware path planning for autonomous vehicles," Transportation Research Part A: Policy and Practice, Elsevier, vol. 134(C), pages 152-163.
    4. Georgios Gioldasis & Antonio Musolesi & Michel Simioni, 2020. "Model uncertainty, nonlinearities and out-of-sample comparison: evidence from international technology diffusion," Working Papers hal-02790523, HAL.
    5. Andrea Bertolini & Massimo Riccaboni, 2021. "Grounding the case for a European approach to the regulation of automated driving: the technology-selection effect of liability rules," European Journal of Law and Economics, Springer, vol. 51(2), pages 243-284, April.
    6. Adam Braima S. Mastor & Abdulaziz S. Alghamdi & Oscar Ngesa & Joseph Mung’atu & Christophe Chesneau & Ahmed Z. Afify, 2023. "The Extended Exponential-Weibull Accelerated Failure Time Model with Application to Sudan COVID-19 Data," Mathematics, MDPI, vol. 11(2), pages 1-26, January.
    7. Khastgir, Siddartha & Brewerton, Simon & Thomas, John & Jennings, Paul, 2021. "Systems Approach to Creating Test Scenarios for Automated Driving Systems," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    8. Yueqi Mao & Qiang Mei & Peng Jing & Ye Zha & Ying Xue & Jiahui Huang & Danning Shao & Pan Luo, 2022. "Factors Affecting the Parental Intention of Using AVs to Escort Children: An Integrated SEM–Hybrid Choice Model Approach," Sustainability, MDPI, vol. 14(18), pages 1-21, September.
    9. Blume, Maximilian & Oberländer, Anna Maria & Röglinger, Maximilian & Rosemann, Michael & Wyrtki, Katrin, 2020. "Ex ante assessment of disruptive threats: Identifying relevant threats before one is disrupted," Technological Forecasting and Social Change, Elsevier, vol. 158(C).
    10. E. Zanini & E. Eastoe & M. J. Jones & D. Randell & P. Jonathan, 2020. "Flexible covariate representations for extremes," Environmetrics, John Wiley & Sons, Ltd., vol. 31(5), August.
    11. Benjamin Säfken & Thomas Kneib, 2020. "Conditional covariance penalties for mixed models," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(3), pages 990-1010, September.
    12. Peng Liu & Run Yang & Zhigang Xu, 2019. "How Safe Is Safe Enough for Self‐Driving Vehicles?," Risk Analysis, John Wiley & Sons, vol. 39(2), pages 315-325, February.
    13. Khamma, Thulasi Ram & Zhang, Yuming & Guerrier, Stéphane & Boubekri, Mohamed, 2020. "Generalized additive models: An efficient method for short-term energy prediction in office buildings," Energy, Elsevier, vol. 213(C).
    14. Winston, Clifford & Karpilow, Quentin, 2017. "A New Route to Increasing Economic Growth: Reducing Highway Congestion with Autonomous Vehicles," Working Papers 03323, George Mason University, Mercatus Center.
    15. Massimiliano Mazzanti & Antonio Musolesi, 2020. "Modeling Green Knowledge Production and Environmental Policies with Semiparametric Panel Data Regression models," SEEDS Working Papers 1420, SEEDS, Sustainability Environmental Economics and Dynamics Studies, revised Sep 2020.
    16. Zoltan Ferenc Magosi & Christoph Wellershaus & Viktor Roland Tihanyi & Patrick Luley & Arno Eichberger, 2022. "Evaluation Methodology for Physical Radar Perception Sensor Models Based on On-Road Measurements for the Testing and Validation of Automated Driving," Energies, MDPI, vol. 15(7), pages 1-20, March.
    17. Andrew Leroux & Junrui Di & Ekaterina Smirnova & Elizabeth J Mcguffey & Quy Cao & Elham Bayatmokhtari & Lucia Tabacu & Vadim Zipunnikov & Jacek K Urbanek & Ciprian Crainiceanu, 2019. "Organizing and Analyzing the Activity Data in NHANES," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 11(2), pages 262-287, July.
    18. Yong Liu & Alan P. Ker, 2021. "Simultaneous borrowing of information across space and time for pricing insurance contracts: An application to rating crop insurance policies," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 88(1), pages 231-257, March.
    19. Stefano Cabras & J. D. Tena, 2023. "Implicit institutional incentives and individual decisions: Causal inference with deep learning models," Managerial and Decision Economics, John Wiley & Sons, Ltd., vol. 44(6), pages 3739-3754, September.
    20. Hudson, John & Orviska, Marta & Hunady, Jan, 2019. "People’s attitudes to autonomous vehicles," Transportation Research Part A: Policy and Practice, Elsevier, vol. 121(C), pages 164-176.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.