IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v34y2018i2p216-224.html
   My bibliography  Save this article

The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial

Author

Listed:
  • Meijering, Jurian Vincent
  • Tobi, Hilde

Abstract

This study examined the effects of feeding back experts’ initial ratings on three Delphi outcome measures: (1) the percentage of items on which experts changed their opinion; (2) the degree to which experts changed their ratings towards the group response; and (3) the increase in the level of agreement among experts. Additionally, two conformity indices were developed. Within a real-world Delphi study, experts were randomly assigned to one of two conditions: either their initial ratings were included in feedback (IN) or excluded from feedback (EX). Results showed that experts in the EX condition changed their opinion relatively more often than experts in the IN condition. Results also suggested that experts in the EX condition changed their ratings to a greater degree towards the group response than experts in the IN condition. No difference between conditions was found regarding the increase in the level of agreement.

Suggested Citation

  • Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    • Handle: RePEc:eee:intfor:v:34:y:2018:i:2:p:216-224
    DOI: 10.1016/j.ijforecast.2017.11.010
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207018300025
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2017.11.010?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Meijering, J.V. & Kampen, J.K. & Tobi, H., 2013. "Quantifying the development of agreement among experts in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1607-1614.
    2. Nicholas Charron & Lewis Dijkstra & Victor Lapuente, 2015. "Mapping the Regional Divide in Europe: A Measure for Assessing Quality of Government in 206 European Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 122(2), pages 315-346, June.
    3. Nicholas Charron & Lewis Dijkstra & Victor Lapuente, 2015. "Erratum to: Mapping the Regional Divide in Europe: A Measure for Assessing Quality of Government in 206 European Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 124(3), pages 1059-1059, December.
    4. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    5. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    6. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    7. von der Gracht, Heiko A., 2012. "Consensus measurement in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 79(8), pages 1525-1536.
    8. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    9. Stanislav Birko & Edward S Dove & Vural Özdemir, 2015. "Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-14, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    2. Flostrand, Andrew & Pitt, Leyland & Bridson, Shannon, 2020. "The Delphi technique in forecasting– A 42-year bibliographic analysis (1975–2017)," Technological Forecasting and Social Change, Elsevier, vol. 150(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    2. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    3. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    4. Nur, Suardi & Burton, Bruce & Bergmann, Ariel, 2023. "Evidence on optimal risk allocation models for Indonesian geothermal projects under PPP contracts," Utilities Policy, Elsevier, vol. 81(C).
    5. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    6. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    7. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    8. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.
    9. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    10. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    11. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    12. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    13. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    14. Mohammed, Sayeed & Desha, Cheryl & Goonetilleke, Ashantha, 2022. "Investigating low-carbon pathways for hydrocarbon-dependent rentier states: Economic transition in Qatar," Technological Forecasting and Social Change, Elsevier, vol. 185(C).
    15. Fritschy, Carolin & Spinler, Stefan, 2019. "The impact of autonomous trucks on business models in the automotive and logistics industry–a Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    16. Förster, Bernadette, 2015. "Technology foresight for sustainable production in the German automotive supplier industry," Technological Forecasting and Social Change, Elsevier, vol. 92(C), pages 237-248.
    17. Marta Salgado & Ana C. L. Vieira & Anália Torres & Mónica D. Oliveira, 2020. "Selecting Indicators to Monitor and Assess Environmental Health in a Portuguese Urban Setting: A Participatory Approach," IJERPH, MDPI, vol. 17(22), pages 1-16, November.
    18. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    19. Menéndez-Caravaca, Eloísa & Bueno, Salvador & Gallego, M. Dolores, 2021. "Exploring the link between free and open source software and the collaborative economy: A Delphi-based scenario for the year 2025," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    20. Tobias Meyer & Heiko A. von der Gracht & Evi Hartmann, 2022. "Technology foresight for sustainable road freight transportation: Insights from a global real‐time Delphi study," Futures & Foresight Science, John Wiley & Sons, vol. 4(1), March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:34:y:2018:i:2:p:216-224. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.