IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v31y2008i2p191-198.html

Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation

Author

Listed:
  • Hung, Hsin-Ling
  • Altschuld, James W.
  • Lee, Yi-Fang

Abstract

Although the Delphi is widely used, research on certain methodological issues is somewhat limited. After a brief introduction to the strengths, limitations, and methodological challenges of the technique, we share our experiences (as well as problems encountered) with an electronic Delphi of educational program evaluation (EPE) in the Asia-Pacific region. The study is described followed by a discussion of the difficulties in participant recruitment and selection, sample size, instrumentation, data collection and analysis and attempts to resolve them. Some of these problems are generic to the Delphi whereas others related to the specifics of the investigation. What we learned should be useful for future Delphis with a similar focus.

Suggested Citation

  • Hung, Hsin-Ling & Altschuld, James W. & Lee, Yi-Fang, 2008. "Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation," Evaluation and Program Planning, Elsevier, vol. 31(2), pages 191-198, May.
    • Handle: RePEc:eee:epplan:v:31:y:2008:i:2:p:191-198
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149-7189(08)00013-X
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    2. George T. Milkovich & Anthony J. Annoni & Thomas A. Mahoney, 1972. "The Use of the Delphi Procedures in Manpower Forecasting," Management Science, INFORMS, vol. 19(4-Part-1), pages 381-388, December.
    3. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peeraer, Jef & Van Petegem, Peter, 2015. "Integration or transformation? Looking in the future of Information and Communication Technology in education in Vietnam," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 47-56.
    2. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165, January.
    3. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    4. Lisheng Weng & Bao-Jie He & Lina Liu & Chenxi Li & Xin Zhang, 2019. "Sustainability Assessment of Cultural Heritage Tourism: Case Study of Pingyao Ancient City in China," Sustainability, MDPI, vol. 11(5), pages 1-16, March.
    5. Rabia Bashir & Aymen Sajjad & Shahid Bashir & Khawaja Fawad Latif & Saman Attiq, 2021. "Project Managers’ Competencies in International Development Projects: A Delphi Study," SAGE Open, , vol. 11(4), pages 21582440211, November.
    6. Toppinen, Anne & Röhr, Axel & Pätäri, Satu & Lähtinen, Katja & Toivonen, Ritva, 2018. "The future of wooden multistory construction in the forest bioeconomy – A Delphi study from Finland and Sweden," Journal of Forest Economics, Elsevier, vol. 31(C), pages 3-10.
    7. Yildirim, Ercan & AR, Ilker Murat & Dabić, Marina & Baki, Birdogan & Peker, Iskender, 2022. "A multi-stage decision making model for determining a suitable innovation structure using an open innovation approach," Journal of Business Research, Elsevier, vol. 147(C), pages 379-391.
    8. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165.
    9. de Loë, Rob C. & Melnychuk, Natalya & Murray, Dan & Plummer, Ryan, 2016. "Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities," Technological Forecasting and Social Change, Elsevier, vol. 104(C), pages 78-88.
    10. Valentina Tudisca & Adriana Valente, 2016. "(English) Design and implementation of an online Delphi study to develop indicators for evidenceinformed policy making (Italiano) Ideazione e implementazione di uno studio Delphi online per lo sviluppo di indicatori di evidence," IRPPS Working Papers 88:2016, National Research Council, Institute for Research on Population and Social Policies.
    11. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    12. Ali Bastas & Kapila Liyanage, 2018. "ISO 9001 and Supply Chain Integration Principles Based Sustainable Development: A Delphi Study," Sustainability, MDPI, vol. 10(12), pages 1-35, December.
    13. V. M. Jayasooriya & A. W. M. Ng & S. Muthukumaran & B. J. C. Perera, 2019. "Multi Criteria Decision Making in Selecting Stormwater Management Green Infrastructure for Industrial Areas Part 1: Stakeholder Preference Elicitation," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(2), pages 627-639, January.
    14. Erol, Ismail & Peker, Iskender & Medeni, Ihsan Tolga & Yuce, Fatma, 2025. "Towards precision dentistry through artificial intelligence and blockchain-based digital Twins: Investigating challenges and solution strategies," Technology in Society, Elsevier, vol. 83(C).
    15. Feil, Alex Sandro & Antunes, Carlos Henggeler & da Silva, Patrícia Pereira & de Castro, Nivalde, 2024. "The critical drivers of the Brazilian electricity sector's transition through 2050: A Delphi study," Utilities Policy, Elsevier, vol. 87(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    2. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    3. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    4. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    5. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    6. Elena Arce, María & Saavedra, Ángeles & Míguez, José L. & Granada, Enrique, 2015. "The use of grey-based methods in multi-criteria decision analysis for the evaluation of sustainable energy systems: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 47(C), pages 924-932.
    7. Georg Aichholzer, 2002. "Das ExpertInnen-Delphi: methodische Grundlagen und Anwendungsfeld ‘Technology Foresight‘ (The Expert Delphi: Methodology and Application in 'Technology Foresight')," ITA manu:scripts 02_01, Institute of Technology Assessment (ITA).
    8. Hayes, Tom, 2007. "Delphi study of the future of marketing of higher education," Journal of Business Research, Elsevier, vol. 60(9), pages 927-931, September.
    9. Marcin Kozak & Olesia Iefremova, 2014. "Implementation Of The Delphi Technique In Finance," "e-Finanse", University of Information Technology and Management, Institute of Financial Research and Analysis, vol. 10(4), pages 36-45, May.
    10. Worrell, James L. & Di Gangi, Paul M. & Bush, Ashley A., 2013. "Exploring the use of the Delphi method in accounting information systems research," International Journal of Accounting Information Systems, Elsevier, vol. 14(3), pages 193-208.
    11. Haarhaus, Tim & Liening, Andreas, 2020. "Building dynamic capabilities to cope with environmental uncertainty: The role of strategic foresight," Technological Forecasting and Social Change, Elsevier, vol. 155(C).
    12. Olga Blasco-Blasco & Pablo A. Pinto-Delacadena & Andrea Vinueza-Cabezas & Vicente Liern, 2025. "Crafting a competency-based questionnaire based on Fuzzy Delphi: a proposal for enhanced hiring process," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(4), pages 3245-3269, August.
    13. Jessica P. Fefer & Sandra De-Urioste Stone & John Daigle & Linda Silka, 2016. "Using the Delphi Technique to Identify Key Elements for Effective and Sustainable Visitor Use Planning Frameworks," SAGE Open, , vol. 6(2), pages 21582440166, April.
    14. Rezaei, Mojtaba & Jafari-Sadeghi, Vahid & Cao, Dongmei & Mahdiraji, Hannan Amoozad, 2021. "Key indicators of ethical challenges in digital healthcare: A combined Delphi exploration and confirmative factor analysis approach with evidence from Khorasan province in Iran," Technological Forecasting and Social Change, Elsevier, vol. 167(C).
    15. Ribeiro, Barbara E. & Quintanilla, Miguel A., 2015. "Transitions in biofuel technologies: An appraisal of the social impacts of cellulosic ethanol using the Delphi method," Technological Forecasting and Social Change, Elsevier, vol. 92(C), pages 53-68.
    16. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    17. Yeh, Duen-Yian & Cheng, Ching-Hsue, 2015. "Recommendation system for popular tourist attractions in Taiwan using Delphi panel and repertory grid techniques," Tourism Management, Elsevier, vol. 46(C), pages 164-176.
    18. Chen-Hua Fu & Chih-Yung Chen, 2021. "A Study on Decision-Making Opinion Exploration in Windows-Based Information Security Monitoring Tool Development," Sustainability, MDPI, vol. 13(7), pages 1-39, March.
    19. Merfeld, Katrin & Wilhelms, Mark-Philipp & Henkel, Sven & Kreutzer, Karin, 2019. "Carsharing with shared autonomous vehicles: Uncovering drivers, barriers and future developments – A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 144(C), pages 66-81.
    20. Delke, Vincent & Schiele, Holger & Buchholz, Wolfgang & Kelly, Stephen, 2023. "Implementing Industry 4.0 technologies: Future roles in purchasing and supply management," Technological Forecasting and Social Change, Elsevier, vol. 196(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:31:y:2008:i:2:p:191-198. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.