IDEAS home Printed from https://ideas.repec.org/a/kap/policy/v52y2019i4d10.1007_s11077-019-09358-y.html
   My bibliography  Save this article

Evaluations as a decent knowledge base? Describing and explaining the quality of the European Commission’s ex-post legislative evaluations

Author

Listed:
  • Stijn Voorst

    (Radboud University)

  • Ellen Mastenbroek

    (Radboud University)

Abstract

Evaluations are a potentially important tool for democratic governments: they provide a basis for accountability and policy learning. To contribute to these key functions, evaluations must be of sufficient methodological quality. However, this quality is threatened by both political influences and technical complexities. This article describes and explains the variance in the quality of ex-post legislative (EPL) evaluations conducted by the European Commission, which is a frontrunner in this realm. A number of potential political and technical explanations of evaluation quality are tested with a unique, self-constructed dataset of 153 EPL evaluations. The results show that the Commission’s EPL evaluations usually apply a robust methodology, while the clarity of their scope, the accuracy of their data and the foundations of their conclusions are problematic. The variance in this quality is mainly explained by the type of evaluator: EPL evaluations conducted by external actors are of higher quality than evaluations conducted internally by the Commission.

Suggested Citation

  • Stijn Voorst & Ellen Mastenbroek, 2019. "Evaluations as a decent knowledge base? Describing and explaining the quality of the European Commission’s ex-post legislative evaluations," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(4), pages 625-644, December.
    • Handle: RePEc:kap:policy:v:52:y:2019:i:4:d:10.1007_s11077-019-09358-y
    DOI: 10.1007/s11077-019-09358-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11077-019-09358-y
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11077-019-09358-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Falk Daviter, 2015. "The political use of knowledge in the policy process," Policy Sciences, Springer;Society of Policy Sciences, vol. 48(4), pages 491-505, December.
    2. Jacopo Torriti, 2010. "Impact Assessment and the Liberalization of the EU Energy Markets: Evidence-Based Policy-Making or Policy-Based Evidence-Making?," Journal of Common Market Studies, Wiley Blackwell, vol. 48, pages 1065-1081, September.
    3. Christian Adam & Yves Steinebach & Christoph Knill, 2018. "Neglected challenges to evidence-based policy-making: the problem of policy accumulation," Policy Sciences, Springer;Society of Policy Sciences, vol. 51(3), pages 269-290, September.
    4. Kaeding, Michael, 2006. "Determinants of Transposition Delay in the European Union," Journal of Public Policy, Cambridge University Press, vol. 26(3), pages 229-253, December.
    5. Lawrence Mead, 2015. "Only connect: Why government often ignores research," Policy Sciences, Springer;Society of Policy Sciences, vol. 48(2), pages 257-272, June.
    6. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    7. Bernard Steunenberg, 2006. "Turning Swift Policy-making into Deadlock and Delay," European Union Politics, , vol. 7(3), pages 293-319, September.
    8. Schwartz, Robert & Mayne, John, 2005. "Assuring the quality of evaluative information: theory and practice," Evaluation and Program Planning, Elsevier, vol. 28(1), pages 1-14.
    9. Jacopo Torriti, 2010. "Impact Assessment and the Liberalization of the EU Energy Markets: Evidence‐Based Policy‐Making or Policy‐Based Evidence‐Making?," Journal of Common Market Studies, Wiley Blackwell, vol. 48(4), pages 1065-1081, September.
    10. Stijn van Voorst & Ellen Mastenbroek, 2017. "Enforcement tool or strategic instrument? The initiation of ex-post legislative evaluations by the European Commission," European Union Politics, , vol. 18(4), pages 640-657, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Steffen Eckhard & Vytautas Jankauskas, 2020. "Explaining the political use of evaluation in international organizations," Policy Sciences, Springer;Society of Policy Sciences, vol. 53(4), pages 667-695, December.
    2. Kupiec, Tomasz, 2022. "Does evaluation quality matter? Quantitative analysis of the use of evaluation findings in the field of cohesion policy in Poland," Evaluation and Program Planning, Elsevier, vol. 93(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stijn van Voorst & Ellen Mastenbroek, 2017. "Enforcement tool or strategic instrument? The initiation of ex-post legislative evaluations by the European Commission," European Union Politics, , vol. 18(4), pages 640-657, December.
    2. Claire A. Dunlop & Martino Maggetti & Claudio M. Radaelli & Duncan Russel, 2012. "The many uses of regulatory impact assessment: A meta‐analysis of EU and UK cases," Regulation & Governance, John Wiley & Sons, vol. 6(1), pages 23-45, March.
    3. Carlos Mendez & John Bachtler, 2017. "Financial Compliance in the European Union: A Cross-National Assessment of Financial Correction Patterns and Causes in Cohesion Policy," Journal of Common Market Studies, Wiley Blackwell, vol. 55(3), pages 569-592, May.
    4. Garrett Ward Richards, 2019. "The Science–Policy Relationship Hierarchy (SPRHi) model of co-production: how climate science organizations have influenced the policy process in Canadian case studies," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(1), pages 67-95, March.
    5. Yuya Kajikawa, 2022. "Reframing evidence in evidence-based policy making and role of bibliometrics: toward transdisciplinary scientometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5571-5585, September.
    6. Giuseppe Munda, 2022. "Qualitative reasoning or quantitative aggregation rules for impact assessment of policy options? A multiple criteria framework," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(5), pages 3259-3277, October.
    7. Miriam Hartlapp & Gerda Falkner, 2009. "Problems of Operationalization and Data in EU Compliance Research," European Union Politics, , vol. 10(2), pages 281-304, June.
    8. Hyensup Shim & Kiyoon Shin, 2021. "Empirical Analysis of Evidence-Based Policymaking in R&D Programmes," Sustainability, MDPI, vol. 14(1), pages 1-15, December.
    9. Esposito, Giovanni & Cicatiello, Lorenzo & Ercolano, Salvatore, 2020. "Reforming railways in the EU: An empirical assessment of liberalisation policies in the European rail freight market," Transportation Research Part A: Policy and Practice, Elsevier, vol. 132(C), pages 606-613.
    10. Egert Juuse & Rainer Kattel, 2014. "Financial Regulation in Estonia," Working papers wpaper57, Financialisation, Economy, Society & Sustainable Development (FESSUD) Project.
    11. Asya Zhelyazkova & René Torenvlied, 2009. "The Time-Dependent Effect of Conflict in the Council on Delays in the Transposition of EU Directives," European Union Politics, , vol. 10(1), pages 35-62, March.
    12. Dimiter Toshkov, 2008. "Embracing European Law," European Union Politics, , vol. 9(3), pages 379-402, September.
    13. Tanja A. Börzel & Tobias Hofmann & Diana Panke, 2011. "Policy Matters But How? Explaining Non-Compliance Dynamics in the EU," KFG Working Papers p0024, Free University Berlin.
    14. Koetter, Michael & Krause, Thomas & Tonzer, Lena, 2019. "Delay determinants of European Banking Union implementation," European Journal of Political Economy, Elsevier, vol. 58(C), pages 1-20.
    15. Schakel, Arjan Hille, 2009. "A Postfunctionalist Theory of Regional Government," MPRA Paper 21596, University Library of Munich, Germany.
    16. Zim Nwokora & Riccardo Pelizzo, 2017. "Measuring Party System Change: A Systems Perspective," Research Africa Network Working Papers 17/048, Research Africa Network (RAN).
    17. Gustav Lidén, 2013. "What about theory? The consequences on a widened perspective of social theory," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(1), pages 213-225, January.
    18. Vargas, Andrés & Sarmiento Erazo, Juan Pablo & Diaz, David, 2020. "Has Cost Benefit Analysis Improved Decisions in Colombia? Evidence from the Environmental Licensing Process," Ecological Economics, Elsevier, vol. 178(C).
    19. Malte Luebker, 2019. "Can the Structure of Inequality Explain Fiscal Redistribution? Revisiting the Social Affinity Hypothesis," LIS Working papers 762, LIS Cross-National Data Center in Luxembourg.
    20. Alberto Arenal & Claudio Feijoo & Ana Moreno & Sergio Ramos & Cristina Armuña, 2021. "Entrepreneurship Policy Agenda in the European Union: A Text Mining Perspective," Review of Policy Research, Policy Studies Organization, vol. 38(2), pages 243-271, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:policy:v:52:y:2019:i:4:d:10.1007_s11077-019-09358-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.