IDEAS home Printed from https://ideas.repec.org/p/ipt/iptwpa/jrc142629.html
   My bibliography  Save this paper

Theory-based evaluation of transformative innovation policies

Author

Listed:

Abstract

"The growing prominence of transformative innovation policy (TIP) necessitates a rethinking of the methodologies, metrics, and practices used in policy monitoring, evaluation, and learning. This article explores the methodological implications of applying theory-based evaluation (TBE) to the assessment of TIP initiatives. TBE emphasizes the identification and validation of causal mechanisms linking policy inputs to outcomes, while accounting for the contextual factors shaping these relationships. We examine key concepts and approaches for empirically testing theories of change, with particular focus on contribution analysis and process tracing as integrative frameworks for combining diverse analytical techniques and data sources. Using the EU Joint Research Centre’s Preparatory Action on place-based TIP as an illustrative case, we demonstrate how these methods can be operationalized in practice. The discussion highlights the critical role of fostering an evaluation culture oriented toward reflexive policy learning. We conclude by underscoring the potential of theory-based evaluation to clarify mechanisms of change, assess the realization of intended outcomes, and strengthen the foundations for continuous policy learning and improvement."

Suggested Citation

  • Janssen Matthijs & Miedzinski Michal & Vankan Arthur, 2025. "Theory-based evaluation of transformative innovation policies," JRC Research Reports JRC142629, Joint Research Centre.
  • Handle: RePEc:ipt:iptwpa:jrc142629
    as

    Download full text from publisher

    File URL: https://publications.jrc.ec.europa.eu/repository/handle/JRC142629
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. JANSSEN Matthijs & WANZENBÖCK Iris & FÜNFSCHILLING Lea & PONTIKAKIS Dimitrios, 2023. "Capacities for transformative innovation in public administrations and governance systems: Evidence from pioneering policy practice," JRC Research Reports JRC131490, Joint Research Centre.
    2. Wise, Emily & Arnold, Erik, 2022. "Evaluating Transformation – what can we learn from the literature?," Papers in Innovation Studies 2022/10, Lund University, CIRCLE - Centre for Innovation Research.
    3. Bipashyee Ghosh & Paula Kivimaa & Matias Ramirez & Johan Schot & Jonas Torrens, 2021. "Transformative outcomes: assessing and reorienting experimentation with transformative innovation policy," Science and Public Policy, Oxford University Press, vol. 48(5), pages 739-756.
    4. Mattia Casula, 2022. "Implementing the transformative innovation policy in the European Union: how does transformative change occur in Member States?," European Planning Studies, Taylor & Francis Journals, vol. 30(11), pages 2178-2204, November.
    5. Jordi Molas-Gallart & Alejandra Boni & Sandro Giachi & Johan Schot, 2021. "A formative approach to the evaluation of Transformative Innovation Policies [The Need for Reflexive Evaluation Approaches in Development Cooperation]," Research Evaluation, Oxford University Press, vol. 30(4), pages 431-442.
    6. Harald Wieser, 2025. "Evaluating in times of paradigmatic change: some reflections on how to conduct ex-post evaluations amidst the transformative turn in research and innovation policy," Research Evaluation, Oxford University Press, vol. 34, pages 59-86.
    7. Rogge, Karoline S. & Reichardt, Kristin, 2016. "Policy mixes for sustainability transitions: An extended concept and framework for analysis," Research Policy, Elsevier, vol. 45(8), pages 1620-1635.
    8. Geels, Frank W., 2002. "Technological transitions as evolutionary reconfiguration processes: a multi-level perspective and a case-study," Research Policy, Elsevier, vol. 31(8-9), pages 1257-1274, December.
    9. Barbara Befani & John Mayne, 2014. "Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation," IDS Bulletin, Blackwell Publishing, vol. 45(6), pages 17-36, November.
    10. MARQUES SANTOS Anabela & COAD Alexander, 2023. "Suggestions for Monitoring and Evaluation of Transformative Innovation Policy," JRC Working Papers on Territorial Modelling and Analysis 2023-03, Joint Research Centre.
    11. Susana Borrás & Sylvia Schwaag Serger, 2022. "The design of transformative research and innovation policy instruments for grand challenges: The policy-nesting perspective [Transformative Innovation Policy Approach to E-waste Management in Ghan," Science and Public Policy, Oxford University Press, vol. 49(5), pages 659-672.
    12. Geels, Frank W., 2022. "Causality and explanation in socio-technical transitions research: Mobilising epistemological insights from the wider social sciences," Research Policy, Elsevier, vol. 51(6).
    13. Erik Arnold, 2004. "Evaluating research and innovation policy: a systems world needs systems evaluations," Research Evaluation, Oxford University Press, vol. 13(1), pages 3-17, April.
    14. Santos, Anabela M. & Coad, Alex, 2023. "Monitoring and evaluation of transformative innovation policy: Suggestions for Improvement," Socio-Economic Planning Sciences, Elsevier, vol. 90(C).
    15. Matthijs J Janssen & Anna Bergek & Joeri H Wesseling, 2022. "Evaluating systemic innovation and transition programmes: Towards a culture of learning," PLOS Sustainability and Transformation, Public Library of Science, vol. 1(3), pages 1-6, March.
    16. Haddad, Carolina R. & Bergek, Anna, 2023. "Towards an integrated framework for evaluating transformative innovation policy," Research Policy, Elsevier, vol. 52(2).
    17. Philippe Larrue & Piret Tõnurist & David Jonason, 2024. "Monitoring and evaluation of mission-oriented innovation policies: From theory to practice," OECD Science, Technology and Industry Working Papers 2024/09, OECD Publishing.
    18. Harald Rohracher & Lars Coenen & Olga Kordas, 2023. "Mission incomplete: Layered practices of monitoring and evaluation in Swedish transformative innovation policy," Science and Public Policy, Oxford University Press, vol. 50(2), pages 336-349.
    19. Diercks, Gijs & Larsen, Henrik & Steward, Fred, 2019. "Transformative innovation policy: Addressing variety in an emerging policy paradigm," Research Policy, Elsevier, vol. 48(4), pages 880-894.
    20. Alejandra Boni & Diana Velasco & Jordi Molas-Gallart & Johan Schot, 2023. "Evaluating transformative innovation policy in a formative way: Insights from Vinnova’s food mission experiment," Research Evaluation, Oxford University Press, vol. 32(3), pages 577-590.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Haddad, Carolina R. & Bergek, Anna, 2023. "Towards an integrated framework for evaluating transformative innovation policy," Research Policy, Elsevier, vol. 52(2).
    2. Laatsit, Mart & Grillitsch, Markus & Fünfschilling, Lea, 2025. "Great expectations: The promises and limits of innovation policy in addressing societal challenges," Research Policy, Elsevier, vol. 54(3).
    3. Paula Kivimaa & Jani Lukkarinen & David Lazarevic, 2023. "Analysis of COVID-19 recovery and resilience policy in Finland: a transformative policy mix approach," Science and Public Policy, Oxford University Press, vol. 50(4), pages 681-694.
    4. Kanger, Laur & Ghosh, Bipashyee & Entsalo, Hanna, 2025. "Integrated framework of intervention points and transformative outcomes for single- and multi-system transitions," Technological Forecasting and Social Change, Elsevier, vol. 216(C).
    5. Santos, Anabela M. & Coad, Alex, 2023. "Monitoring and evaluation of transformative innovation policy: Suggestions for Improvement," Socio-Economic Planning Sciences, Elsevier, vol. 90(C).
    6. repec:oup:rseval:v:32:y:2024:i:3:p:577-590. is not listed on IDEAS
    7. Harald Rohracher & Lars Coenen & Olga Kordas, 2023. "Mission incomplete: Layered practices of monitoring and evaluation in Swedish transformative innovation policy," Science and Public Policy, Oxford University Press, vol. 50(2), pages 336-349.
    8. Wise, Emily & Arnold, Erik, 2022. "Evaluating Transformation – what can we learn from the literature?," Papers in Innovation Studies 2022/10, Lund University, CIRCLE - Centre for Innovation Research.
    9. Maria Tomai & Shyama V. Ramani & George Papachristos, 2024. "How Can We Design Policy Better? Frameworks and Approaches for Sustainability Transitions," Sustainability, MDPI, vol. 16(2), pages 1-22, January.
    10. Joeri Wesseling & Nick Meijerhof, 2023. "Towards a Mission-oriented Innovation Systems (MIS) approach, application for Dutch sustainable maritime shipping," PLOS Sustainability and Transformation, Public Library of Science, vol. 2(8), pages 1-28, August.
    11. Weber, Karl Matthias & Giesecke, Susanne & Havas, Attila & Schartinger, Doris & Albiez, Andreas & Horak, Sophia & Blind, Knut & Bodenheimer, Miriam & Daimer, Stephanie & Shi, Liu & Stadler, Maria & Sc, 2024. "Social innovation - (Accompanying) instrument for addressing societal challenges?," Studien zum deutschen Innovationssystem 10-2024, Expertenkommission Forschung und Innovation (EFI) - Commission of Experts for Research and Innovation, Berlin.
    12. Chen, Kaihua & Xue, Zehua & Guo, Rui & Ning, Lutao, 2025. "The holistic role of multi-level government in transformative innovation process: Theoretical framework and evidence from China," Technovation, Elsevier, vol. 139(C).
    13. Pontikakis Dimitrios & Papachristos Georgios & Janssen Matthijs & Norlen Hedvig & Miedzinski Michal, 2025. "System Dynamics for System Innovation," JRC Research Reports JRC143019, Joint Research Centre.
    14. Janssen, Matthijs & Wanzenböck, Iris & Fünfschilling, Lea & Pontinakis, Dimitris, 2023. "Governance arrangements for the implementation of transformative innovation policy: Insights from a comparative case study," Papers in Innovation Studies 2023/13, Lund University, CIRCLE - Centre for Innovation Research.
    15. Laatsit, Mart & Grillitsch, Markus & Fünfschilling, Lea, 2022. "Great expectations: the promises and limits of innovation policy in addressing societal challenges," Papers in Innovation Studies 2022/9, Lund University, CIRCLE - Centre for Innovation Research.
    16. Švarc, Jadranka & Dabić, Marina, 2021. "Transformative innovation policy or how to escape peripheral policy paradox in European research peripheral countries," Technology in Society, Elsevier, vol. 67(C).
    17. Alyousef, Mohammed & Belaid, Fateh & Almubarak, Norah & Almulhim, Tarifa, 2025. "Mapping Saudi Arabia's low emissions transition path by 2060: An input-output analysis," Technological Forecasting and Social Change, Elsevier, vol. 211(C).
    18. Nina Savela & Jarkko Levänen & Sara Lindeman & Nnenesi Kgabi & Heikki Koivisto & Meri Olenius & Samuel John & Damas Mashauri & Minna M. Keinänen-Toivola, 2020. "Rapid Urbanization and Infrastructure Pressure: Comparing the Sustainability Transition Potential of Water and Energy Regimes in Namibia," World, MDPI, vol. 1(2), pages 1-18, July.
    19. Chunhong Sheng & Yun Cao & Bing Xue, 2018. "Residential Energy Sustainability in China and Germany: The Impact of National Energy Policy System," Sustainability, MDPI, vol. 10(12), pages 1-18, December.
    20. Baxter, David & Trott, Paul & Ellwood, Paul, 2023. "Reconceptualising innovation failure," Research Policy, Elsevier, vol. 52(7).
    21. Holmes, Torik, 2024. "We need more network capacity! But why do we get it? Infrastructural refraction and the contingencies of network growth and energy transitions," Technological Forecasting and Social Change, Elsevier, vol. 205(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ipt:iptwpa:jrc142629. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Publication Officer (email available below). General contact details of provider: https://edirc.repec.org/data/ipjrces.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.