IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v228y2022ics0951832022004252.html
   My bibliography  Save this article

Large-scale group decision-making to facilitate inter-rater reliability of human-factors analysis for the railway system

Author

Listed:
  • Zhou, Jian-Lan
  • Tu, Ren-Fang
  • Xiao, Hai

Abstract

The interdependency analysis of human factors usually needs to identify and label factors from accident/incident reports using a binary framework. The labeling procedure involves enormous knowledge and requires a large group of raters, but the raters’ opinions usually highly contradict each other. Thus, inter-rater reliability should be implemented to improve the labeling consistency. However, the inter-rater reliability coefficient is commonly computed on a category-by-category basis, which is inefficient especially when there are many raters. To overcome this deficiency, a feedback-autonomy-based consensus model is proposed to determine the inter-rater reliability of human-factors labeling results involving a large group of raters. The proposed model can manage various non-cooperative behaviors and provide feedback adjustment suggestions to raters for the labeling procedure. Notably, the autonomy mechanism allows raters to freely choose and adjust labels referring to the reference opinions without causing over-adjustment issue. The inter-rater reliability and the interdependency results are performed on 279 accident/incident reports. We apply a commercial tool to obtain the reference results, which are consistent with the unanimous group opinions. Besides, other comparative studies clarify the advantages of the proposed consensus model. The proposed model is convenient for reducing conflicts among raters that usually exist in human-factors analysis.

Suggested Citation

  • Zhou, Jian-Lan & Tu, Ren-Fang & Xiao, Hai, 2022. "Large-scale group decision-making to facilitate inter-rater reliability of human-factors analysis for the railway system," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    • Handle: RePEc:eee:reensy:v:228:y:2022:i:c:s0951832022004252
    DOI: 10.1016/j.ress.2022.108806
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832022004252
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2022.108806?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yoon, Young Sik & Ham, Dong-Han & Yoon, Wan Chul, 2016. "Application of activity theory to analysis of human-related accidents: Method and case studies," Reliability Engineering and System Safety, Elsevier, vol. 150(C), pages 22-34.
    2. Cai, Wei & Zhao, Jingyi & Zhu, Ming, 2020. "A real time methodology of cluster-system theory-based reliability estimation using k-means clustering," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    3. Kim, Dong San & Baek, Dong Hyun & Yoon, Wan Chul, 2010. "Development and evaluation of a computer-aided system for analyzing human error in railway operations," Reliability Engineering and System Safety, Elsevier, vol. 95(2), pages 87-98.
    4. Xiaohong Chen & Weiwei Zhang & Xuanhua Xu & Wenzhi Cao, 2022. "Managing Group Confidence and Consensus in Intuitionistic Fuzzy Large Group Decision-Making Based on Social Media Data Mining," Group Decision and Negotiation, Springer, vol. 31(5), pages 995-1023, October.
    5. Bevilacqua, Maurizio & Ciarapica, Filippo Emanuele, 2018. "Human factor risk management in the process industry: A case study," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 149-159.
    6. Lo, Huai-Wei & Liou, James J.H. & Huang, Chun-Nen & Chuang, Yen-Ching, 2019. "A novel failure mode and effect analysis model for machine tool risk analysis," Reliability Engineering and System Safety, Elsevier, vol. 183(C), pages 173-183.
    7. Xia-Zhong Zheng & Fei Wang & Jian-Lan Zhou, 2017. "A Hybrid Approach for Evaluating Faulty Behavior Risk of High-Risk Operations Using ANP and Evidence Theory," Mathematical Problems in Engineering, Hindawi, vol. 2017, pages 1-16, August.
    8. Abrishami, Shokoufeh & Khakzad, Nima & Hosseini, Seyed Mahmoud & van Gelder, Pieter, 2020. "BN-SLIM: A Bayesian Network methodology for human reliability assessment based on Success Likelihood Index Method (SLIM)," Reliability Engineering and System Safety, Elsevier, vol. 193(C).
    9. Liu, Hu-Chen & Li, Zhaojun & Zhang, Jian-Qing & You, Xiao-Yue, 2018. "A large group decision making approach for dependence assessment in human reliability analysis," Reliability Engineering and System Safety, Elsevier, vol. 176(C), pages 135-144.
    10. Kaptan, Mehmet & Uğurlu, Özkan & Wang, Jin, 2021. "The effect of nonconformities encountered in the use of technology on the occurrence of collision, contact and grounding accidents," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    11. Dindar, Serdar & Kaewunruen, Sakdirat & An, Min, 2020. "Bayesian network-based human error reliability assessment of derailments," Reliability Engineering and System Safety, Elsevier, vol. 197(C).
    12. Zhou, Jian-Lan & Lei, Yi, 2020. "A slim integrated with empirical study and network analysis for human error assessment in the railway driving process," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    13. Laumann, Karin & Rasmussen, Martin, 2016. "Suggested improvements to the definitions of Standardized Plant Analysis of Risk-Human Reliability Analysis (SPAR-H) performance shaping factors, their levels and multipliers and the nominal tasks," Reliability Engineering and System Safety, Elsevier, vol. 145(C), pages 287-300.
    14. Liu, Peng & Qiu, Yongping & Hu, Juntao & Tong, Jiejuan & Zhao, Jun & Li, Zhizhong, 2020. "Expert judgments for performance shaping Factors’ multiplier design in human reliability analysis," Reliability Engineering and System Safety, Elsevier, vol. 194(C).
    15. Evans, Andrew W., 2013. "The economics of railway safety," Research in Transportation Economics, Elsevier, vol. 43(1), pages 137-147.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Catelani, Marcantonio & Ciani, Lorenzo & Guidi, Giulia & Patrizi, Gabriele, 2021. "An enhanced SHERPA (E-SHERPA) method for human reliability analysis in railway engineering," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    2. Zhou, Jian-Lan & Yu, Ze-Tai & Xiao, Ren-Bin, 2022. "A large-scale group Success Likelihood Index Method to estimate human error probabilities in the railway driving process," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    3. Zhou, Jian-Lan & Lei, Yi, 2020. "A slim integrated with empirical study and network analysis for human error assessment in the railway driving process," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    4. Liu, Hu-Chen & Wang, Jing-Hui & Zhang, Ling & Zhang, Qi-Zhen, 2022. "New success likelihood index model for large group human reliability analysis considering noncooperative behaviors and social network," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    5. Liu, Jianqiao & Zou, Yanhua & Wang, Wei & Zio, Enrico & Yuan, Chengwei & Wang, Taorui & Jiang, Jianjun, 2022. "A Bayesian belief network framework for nuclear power plant human reliability analysis accounting for dependencies among performance shaping factors," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    6. Sezer, Sukru Ilke & Akyuz, Emre & Arslan, Ozcan, 2022. "An extended HEART Dempster–Shafer evidence theory approach to assess human reliability for the gas freeing process on chemical tankers," Reliability Engineering and System Safety, Elsevier, vol. 220(C).
    7. Liu, Zhichen & Li, Ying & Zhang, Zhaoyi & Yu, Wenbo, 2022. "A new evacuation accessibility analysis approach based on spatial information," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    8. Abrishami, Shokoufeh & Khakzad, Nima & Hosseini, Seyed Mahmoud, 2020. "A data-based comparison of BN-HRA models in assessing human error probability: An offshore evacuation case study," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    9. Chen, Yuanjiang & Feng, Wei & Jiang, Zhiqiang & Duan, Lingling & Cheng, Shuangyi, 2021. "An accident causation model based on safety information cognition and its application," Reliability Engineering and System Safety, Elsevier, vol. 207(C).
    10. Rungskunroch, Panrawee & Jack, Anson & Kaewunruen, Sakdirat, 2021. "Benchmarking on railway safety performance using Bayesian inference, decision tree and petri-net techniques based on long-term accidental data sets," Reliability Engineering and System Safety, Elsevier, vol. 213(C).
    11. Ewa DUDEK & Karolina KRZYKOWSKA-PIOTROWSKA & Mirosław SIERGIEJCZYK, 2020. "Risk Management In (Air) Transport With Exemplary Risk Analysis Based On The Tolerability Matrix," Transport Problems, Silesian University of Technology, Faculty of Transport, vol. 15(2), pages 143-156, June.
    12. Tang, Ming & Liao, Huchang, 2021. "Failure mode and effect analysis considering the fairness-oriented consensus of a large group with core-periphery structure," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    13. Maurizio Bevilacqua & Eleonora Bottani & Filippo Emanuele Ciarapica & Francesco Costantino & Luciano Di Donato & Alessandra Ferraro & Giovanni Mazzuto & Andrea Monteriù & Giorgia Nardini & Marco Orten, 2020. "Digital Twin Reference Model Development to Prevent Operators’ Risk in Process Plants," Sustainability, MDPI, vol. 12(3), pages 1-17, February.
    14. Dindar, Serdar & Kaewunruen, Sakdirat & An, Min, 2022. "A hierarchical Bayesian-based model for hazard analysis of climate effect on failures of railway turnout components," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    15. Wang, Tianzhe & Chen, Zequan & Li, Guofa & He, Jialong & Liu, Chao & Du, Xuejiao, 2024. "A novel method for high-dimensional reliability analysis based on activity score and adaptive Kriging," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    16. Ji, Chenyi & Su, Xing & Qin, Zhongfu & Nawaz, Ahsan, 2022. "Probability Analysis of Construction Risk based on Noisy-or Gate Bayesian Networks," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    17. Felipe Aguirre & Mohamed Sallak & Walter Schön & Fabien Belmonte, 2013. "Application of evidential networks in quantitative analysis of railway accidents," Journal of Risk and Reliability, , vol. 227(4), pages 368-384, August.
    18. Jang, Inseok & Kim, Yochan & Park, Jinkyun, 2021. "Investigating the Effect of Task Complexity on the Occurrence of Human Errors observed in a Nuclear Power Plant Full-Scope Simulator," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    19. Burns, Kevin & Bonaceto, Craig, 2020. "An empirically benchmarked human reliability analysis of general aviation," Reliability Engineering and System Safety, Elsevier, vol. 194(C).
    20. Zhang, Yan & Wang, Yu-Hao & Zhao, Xu & Tong, Rui-Peng, 2023. "Dynamic probabilistic risk assessment of emergency response for intelligent coal mining face system, case study: Gas overrun scenario," Resources Policy, Elsevier, vol. 85(PB).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:228:y:2022:i:c:s0951832022004252. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.