IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v108y2025ics0149718924001113.html
   My bibliography  Save this article

Anticipatory evaluation. How to incorporate an anticipatory technique into a theory-driven evaluation process. Results of application in a case study

Author

Listed:
  • Pacheco – Troisi, Mariangel
  • García-Melón, Mónica
  • Jiménez-Sáez, Fernando

Abstract

In recent years, there has been increased focus on strategic learning from impact analysis, including in the field of science, technology, and innovation. In this paper, we propose combining techniques from two fields of study. Firstly, we adopt the approach of addressing impact through productive interactions between science and society, and secondly, we incorporate an anticipatory dimension by integrating game analysis involving key actors. Through a theory-driven evaluation design, we consider expected impacts as promises of the future. Within an anticipatory perspective, the future can be shaped by the interactions among different actors in the present. In this article, we apply this approach step-by-step to a research institute program in Uruguay. We demonstrate how the achieved results offer strategic insights to the program manager for anticipating and attaining the desired impacts. Additionally, we provide summative inputs for accountability using a flexible technique applicable at any stage of the program life cycle. The article concludes with a discussion of the advantages and disadvantages compared to other techniques, along with lessons learned that may benefit other evaluators seeking to replicate this approach. Furthermore, we explore potential extensions and opportunities for further improvement in this research.

Suggested Citation

  • Pacheco – Troisi, Mariangel & García-Melón, Mónica & Jiménez-Sáez, Fernando, 2025. "Anticipatory evaluation. How to incorporate an anticipatory technique into a theory-driven evaluation process. Results of application in a case study," Evaluation and Program Planning, Elsevier, vol. 108(C).
    • Handle: RePEc:eee:epplan:v:108:y:2025:i:c:s0149718924001113
    DOI: 10.1016/j.evalprogplan.2024.102509
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718924001113
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2024.102509?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    2. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    3. Laura Meagher & Catherine Lyall & Sandra Nutley, 2008. "Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research," Research Evaluation, Oxford University Press, vol. 17(3), pages 163-173, September.
    4. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    5. Sarah de Rijcke & Paul F. Wouters & Alex D. Rushforth & Thomas P. Franssen & Björn Hammarfelt, 2016. "Evaluation practices and effects of indicator use—a literature review," Research Evaluation, Oxford University Press, vol. 25(2), pages 161-169.
    6. Pablo D’Este & Irene Ramos-Vielba & Richard Woolley & Nabil Amara, 2018. "How do researchers generate scientific and societal impacts? Toward an analytical and operational framework," Science and Public Policy, Oxford University Press, vol. 45(6), pages 752-763.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    2. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    3. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    4. Rosa Kuipers-Dirven & Matthijs Janssen & Jarno Hoekman, 2023. "Assessing university policies for enhancing societal impact of academic research: A multicriteria mapping approach," Research Evaluation, Oxford University Press, vol. 32(2), pages 371-383.
    5. repec:oup:rseval:v:32:y:2024:i:2:p:273-285. is not listed on IDEAS
    6. repec:osf:socarx:9n347_v1 is not listed on IDEAS
    7. Irene Ramos-Vielba & Nicolas Robinson-Garcia & Richard Woolley, 2022. "A value creation model from science-society interconnections: Archetypal analysis combining publications, survey and altmetric data," PLOS ONE, Public Library of Science, vol. 17(6), pages 1-23, June.
    8. Ohid Yaqub & Dmitry Malkov & Josh Siepel, 2023. "How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework," Research Evaluation, Oxford University Press, vol. 32(2), pages 273-285.
    9. Frank J. Rijnsoever & Laurens K. Hessels, 2021. "How academic researchers select collaborative research projects: a choice experiment," The Journal of Technology Transfer, Springer, vol. 46(6), pages 1917-1948, December.
    10. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    11. Thomas, Duncan Andrew & Ramos-Vielba, Irene, 2022. "Reframing study of research(er) funding towards configurations and trails," SocArXiv uty2v, Center for Open Science.
    12. John P. Nelson & Barry Bozeman & Stuart Bretschneider & Spencer L. Lindsay, 2024. "How do academic public administration and public policy researchers affect policymaking? Functional groupings from survey data," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(1), pages 65-93, January.
    13. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    14. repec:oup:rseval:v:32:y:2024:i:2:p:348-355. is not listed on IDEAS
    15. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    16. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    17. Thomas Woodson & Sophia Boutilier, 2023. "From intent to impact—The decline of broader impacts throughout an NSF project life cycle," Research Evaluation, Oxford University Press, vol. 32(2), pages 348-355.
    18. repec:oup:rseval:v:32:y:2024:i:2:p:371-383. is not listed on IDEAS
    19. Dorta-González, Pablo & Rodríguez-Caro, Alejandro & Dorta-González, María Isabel, 2024. "Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton," Journal of Informetrics, Elsevier, vol. 18(3).
    20. Genowefa Blundo-Canto & Bernard Triomphe & Guy Faure & Danielle Barret & Aurelle de Romemont & Etienne Hainzelin, 2019. "Building a culture of impact in an international agricultural research organization: Process and reflective learning," Research Evaluation, Oxford University Press, vol. 28(2), pages 136-144.
    21. Cole, Stroma & Wardana, Agung & Dharmiasih, Wiwik, 2021. "Making an impact on Bali's water crisis: Research to mobilize NGOs, the tourism industry and policy makers," Annals of Tourism Research, Elsevier, vol. 87(C).
    22. Ciarli, Tommaso & Ràfols, Ismael, 2019. "The relation between research priorities and societal demands: The case of rice," Research Policy, Elsevier, vol. 48(4), pages 949-967.
    23. Rau, Henrike & Goggins, Gary & Fahy, Frances, 2018. "From invisibility to impact: Recognising the scientific and societal relevance of interdisciplinary sustainability research," Research Policy, Elsevier, vol. 47(1), pages 266-276.
    24. Stefan P L de Jong & Corina Balaban & Maria Nedeva, 2022. "From ‘productive interactions’ to ‘enabling conditions’: The role of organizations in generating societal impact of academic research [One Size Does Not Fit All! New Perspectives on the University ," Science and Public Policy, Oxford University Press, vol. 49(4), pages 643-645.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:108:y:2025:i:c:s0149718924001113. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.