IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0135162.html
   My bibliography  Save this article

Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation

Author

Listed:
  • Stanislav Birko
  • Edward S Dove
  • Vural Özdemir

Abstract

The extent of consensus (or the lack thereof) among experts in emerging fields of innovation can serve as antecedents of scientific, societal, investor and stakeholder synergy or conflict. Naturally, how we measure consensus is of great importance to science and technology strategic foresight. The Delphi methodology is a widely used anonymous survey technique to evaluate consensus among a panel of experts. Surprisingly, there is little guidance on how indices of consensus can be influenced by parameters of the Delphi survey itself. We simulated a classic three-round Delphi survey building on the concept of clustered consensus/dissensus. We evaluated three study characteristics that are pertinent for design of Delphi foresight research: (1) the number of survey questions, (2) the sample size, and (3) the extent to which experts conform to group opinion (the Group Conformity Index) in a Delphi study. Their impacts on the following nine Delphi consensus indices were then examined in 1000 simulations: Clustered Mode, Clustered Pairwise Agreement, Conger’s Kappa, De Moivre index, Extremities Version of the Clustered Pairwise Agreement, Fleiss’ Kappa, Mode, the Interquartile Range and Pairwise Agreement. The dependency of a consensus index on the Delphi survey characteristics was expressed from 0.000 (no dependency) to 1.000 (full dependency). The number of questions (range: 6 to 40) in a survey did not have a notable impact whereby the dependency values remained below 0.030. The variation in sample size (range: 6 to 50) displayed the top three impacts for the Interquartile Range, the Clustered Mode and the Mode (dependency = 0.396, 0.130, 0.116, respectively). The Group Conformity Index, a construct akin to measuring stubbornness/flexibility of experts’ opinions, greatly impacted all nine Delphi consensus indices (dependency = 0.200 to 0.504), except the Extremity CPWA and the Interquartile Range that were impacted only beyond the first decimal point (dependency = 0.087 and 0.083, respectively). Scholars in technology design, foresight research and future(s) studies might consider these new findings in strategic planning of Delphi studies, for example, in rational choice of consensus indices and sample size, or accounting for confounding factors such as experts’ variable degrees of conformity (stubbornness/flexibility) in modifying their opinions.

Suggested Citation

  • Stanislav Birko & Edward S Dove & Vural Özdemir, 2015. "Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-14, August.
    • Handle: RePEc:plo:pone00:0135162
    DOI: 10.1371/journal.pone.0135162
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0135162
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0135162&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0135162?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Alessandro Di Mare & Vito Latora, 2007. "Opinion Formation Models Based On Game Theory," International Journal of Modern Physics C (IJMPC), World Scientific Publishing Co. Pte. Ltd., vol. 18(09), pages 1377-1395.
    2. Matthijs Warrens, 2010. "Inequalities between multi-rater kappas," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 4(4), pages 271-286, December.
    3. von der Gracht, Heiko A., 2012. "Consensus measurement in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 79(8), pages 1525-1536.
    4. Meijering, J.V. & Kampen, J.K. & Tobi, H., 2013. "Quantifying the development of agreement among experts in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1607-1614.
    5. Ulucanlar, S. & Faulkner, A. & Peirce, S. & Elwyn, G., 2013. "Technology identity: The role of sociotechnical representations in the adoption of medical devices," Social Science & Medicine, Elsevier, vol. 98(C), pages 95-105.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tobias Meyer & Heiko A. von der Gracht & Evi Hartmann, 2022. "Technology foresight for sustainable road freight transportation: Insights from a global real‐time Delphi study," Futures & Foresight Science, John Wiley & Sons, vol. 4(1), March.
    2. M. K. Dhillon & P. M. Rafi-Ul-Shan & H. Amar & F. Sher & S. Ahmed, 2023. "Flexible Green Supply Chain Management in Emerging Economies: A Systematic Literature Review," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 24(1), pages 1-28, March.
    3. Mohammed, Sayeed & Desha, Cheryl & Goonetilleke, Ashantha, 2022. "Investigating low-carbon pathways for hydrocarbon-dependent rentier states: Economic transition in Qatar," Technological Forecasting and Social Change, Elsevier, vol. 185(C).
    4. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    5. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    6. Yousif Jaleel & Mohd Saidin Misnan & Mohamad Zahierruden Ismail, 2024. "Toward Sustainable Housing Construction in Iraq," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(5), pages 580-595, May.
    7. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    8. Chia-Hui Hung & Yu-Ming Wang & Cheng-Yi Huang & Chung-Hui Lin, 2022. "Developing of Multidimensional Perspectives Checklist of Professionalism for Undergraduate Occupational Therapy Students in Assistive Technology Service: Delphi Study," IJERPH, MDPI, vol. 19(12), pages 1-13, June.
    9. Zhangyuan He & Hans-Dietrich Haasis, 2020. "A Theoretical Research Framework of Future Sustainable Urban Freight Transport for Smart Cities," Sustainability, MDPI, vol. 12(5), pages 1-28, March.
    10. Feil, Alex Sandro & Antunes, Carlos Henggeler & da Silva, Patrícia Pereira & de Castro, Nivalde, 2024. "The critical drivers of the Brazilian electricity sector's transition through 2050: A Delphi study," Utilities Policy, Elsevier, vol. 87(C).
    11. repec:osf:osfxxx:4ufzg_v1 is not listed on IDEAS
    12. Wen-Cheng Huang & Yi-Ying Lee, 2016. "Strategic Planning for Land Use under Extreme Climate Changes: A Case Study in Taiwan," Sustainability, MDPI, vol. 8(1), pages 1-17, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    2. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    3. Marta Salgado & Ana C. L. Vieira & Anália Torres & Mónica D. Oliveira, 2020. "Selecting Indicators to Monitor and Assess Environmental Health in a Portuguese Urban Setting: A Participatory Approach," IJERPH, MDPI, vol. 17(22), pages 1-16, November.
    4. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    5. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    6. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    7. Nur, Suardi & Burton, Bruce & Bergmann, Ariel, 2023. "Evidence on optimal risk allocation models for Indonesian geothermal projects under PPP contracts," Utilities Policy, Elsevier, vol. 81(C).
    8. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.
    9. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    10. Nibedita Mukherjee & Jean Huge & Farid Dahdouh-Guebas & Nico Koedam, 2014. "Ecosystem service valuations of mangrove ecosystems to inform decision making and future valuation exercises," ULB Institutional Repository 2013/217963, ULB -- Universite Libre de Bruxelles.
    11. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    12. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    13. Petreski Marjan & Petreski Blagica & Tumanoska Despina & Narazani Edlira & Kazazi Fatush & Ognjanov Galjina & Jankovic Irena & Mustafa Arben & Kochovska Tereza, 2017. "The Size and Effects of Emigration and Remittances in the Western Balkans. A Forecasting Based on a Delphi Process," Comparative Southeast European Studies, De Gruyter, vol. 65(4), pages 679-695, December.
    14. Roudgarmi, Pezhman & Mahdiraji, Mohammad Taghi Amoozadeh, 2020. "Current Challenges of Laws for Preservation of Forest and Rangeland, Iran," Land Use Policy, Elsevier, vol. 99(C).
    15. Adriana Luciano & Federica Pascale & Francesco Polverino & Alison Pooley, 2020. "Measuring Age-Friendly Housing: A Framework," Sustainability, MDPI, vol. 12(3), pages 1-35, January.
    16. Johannes I. F. Henning & Henry Jordaan, 2016. "Determinants of Financial Sustainability for Farm Credit Applications—A Delphi Study," Sustainability, MDPI, vol. 8(1), pages 1-15, January.
    17. Matthijs J. Warrens, 2021. "Kappa coefficients for dichotomous-nominal classifications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(1), pages 193-208, March.
    18. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    19. Paulina Tsvetkova & Carla Sousa & Daniel Beiderbeck & Aneta M. Kochanowicz & Branislav Gerazov & May Agius & Tomasz Przybyła & Merita Hoxha & Alan H. Tkaczyk, 2024. "International Perspectives on Assistive Technologies for Autism and Intellectual Disabilities: Findings from a Delphi Study," Disabilities, MDPI, vol. 4(4), pages 1-18, December.
    20. Saskia Crucke & Adelien Decramer, 2016. "The Development of a Measurement Instrument for the Organizational Performance of Social Enterprises," Sustainability, MDPI, vol. 8(2), pages 1-30, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0135162. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.