IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v140y2019icp296-305.html
   My bibliography  Save this article

Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback

Author

Listed:
  • Kawamoto, Carlos Tadao
  • Wright, James Terence Coulter
  • Spers, Renata Giovinazzo
  • de Carvalho, Daniel Estima

Abstract

The main goal of this study is to investigate an alternative feedback for Delphi-like applications, based on a subgroup of elite panellists selected from their perception of questions' easiness, in order to increase the results' accuracy. The use of declared easiness perception is an old idea appearing in the earliest studies by Rand Corporation members, but rarely explored in recent publications. We ran an experiment with 79 students and 43 questions in 2015 and 2016, where the treatment subgroup received the assembled elite feedback, and the control subgroup received statistics based on all panellists. The results did not show a statistically significant predominance of higher accuracy with feedback based on the elite subgroup, although correlation between accuracy and easiness perception was not negligible.

Suggested Citation

  • Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    • Handle: RePEc:eee:tefoso:v:140:y:2019:i:c:p:296-305
    DOI: 10.1016/j.techfore.2018.12.020
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162518311405
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2018.12.020?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yaniv, Ilan, 2011. "Group diversity and decision quality: Amplification and attenuation of the framing effect," International Journal of Forecasting, Elsevier, vol. 27(1), pages 41-49, January.
    2. Flostrand, Andrew, 2017. "Finding the future: Crowdsourcing versus the Delphi technique," Business Horizons, Elsevier, vol. 60(2), pages 229-236.
    3. Yaniv, Ilan, 2011. "Group diversity and decision quality: Amplification and attenuation of the framing effect," International Journal of Forecasting, Elsevier, vol. 27(1), pages 41-49.
    4. Bolger, Fergus & Wright, George, 2017. "Use of expert knowledge to anticipate the future: Issues, analysis and directions," International Journal of Forecasting, Elsevier, vol. 33(1), pages 230-243.
    5. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2012. "Experimental methods: Between-subject and within-subject design," Journal of Economic Behavior & Organization, Elsevier, vol. 81(1), pages 1-8.
    6. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    7. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    8. Vernon L. Smith, 1962. "An Experimental Study of Competitive Market Behavior," Journal of Political Economy, University of Chicago Press, vol. 70, pages 111-111.
    9. Kauko, Karlo & Palmroos, Peter, 2014. "The Delphi method in forecasting financial markets— An experimental study," International Journal of Forecasting, Elsevier, vol. 30(2), pages 313-327.
    10. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    11. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    12. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    13. Marcin Kozak & Olesia Iefremova, 2014. "Implementation Of The Delphi Technique In Finance," "e-Finanse", University of Information Technology and Management, Institute of Financial Research and Analysis, vol. 10(4), pages 36-45, May.
    14. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    15. von der Gracht, Heiko A., 2012. "Consensus measurement in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 79(8), pages 1525-1536.
    16. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mauksch, Stefanie & von der Gracht, Heiko A. & Gordon, Theodore J., 2020. "Who is an expert for foresight? A review of identification methods," Technological Forecasting and Social Change, Elsevier, vol. 154(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    2. Schlecht, Laura & Schneider, Sabrina & Buchwald, Arne, 2021. "The prospective value creation potential of Blockchain in business models: A delphi study," Technological Forecasting and Social Change, Elsevier, vol. 166(C).
    3. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    4. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    5. Marcin Kozak & Olesia Iefremova, 2014. "Implementation Of The Delphi Technique In Finance," "e-Finanse", University of Information Technology and Management, Institute of Financial Research and Analysis, vol. 10(4), pages 36-45, May.
    6. Bonaccorsi, Andrea & Apreda, Riccardo & Fantoni, Gualtiero, 2020. "Expert biases in technology foresight. Why they are a problem and how to mitigate them," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    7. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    8. von Briel, Frederik, 2018. "The future of omnichannel retail: A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 132(C), pages 217-229.
    9. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    10. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    11. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    12. Nur, Suardi & Burton, Bruce & Bergmann, Ariel, 2023. "Evidence on optimal risk allocation models for Indonesian geothermal projects under PPP contracts," Utilities Policy, Elsevier, vol. 81(C).
    13. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    14. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.
    15. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    16. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    17. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    18. Antonelli, Marta & Basile, Linda & Gagliardi, Francesca & Isernia, Pierangelo, 2022. "The future of the Mediterranean agri-food systems: Trends and perspectives from a Delphi survey," Land Use Policy, Elsevier, vol. 120(C).
    19. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    20. Volkmar, Gioia & Fischer, Peter M. & Reinecke, Sven, 2022. "Artificial Intelligence and Machine Learning: Exploring drivers, barriers, and future developments in marketing management," Journal of Business Research, Elsevier, vol. 149(C), pages 599-614.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:140:y:2019:i:c:p:296-305. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.