IDEAS home Printed from https://ideas.repec.org/a/kap/itaxpf/v32y2025i2d10.1007_s10797-024-09875-4.html
   My bibliography  Save this article

Evidence-based policy or beauty contest? An LLM-based meta-analysis of EU cohesion policy evaluations

Author

Listed:
  • Zareh Asatryan

    (ZEW Mannheim)

  • Carlo Birkholz

    (University of Mannheim)

  • Friedrich Heinemann

    (University of Heidelberg)

Abstract

Independent and high-quality evaluations of government policies are an important input for designing evidence-based policy. Institutional frictions and lack of incentives to write such evaluations, on the other hand, carry the risk of turning the system into a costly beauty contest. We study one of the most advanced markets of policy evaluations in the world, the evaluations of EU Cohesion Policy interventions by the EU Member States. We use a large language model to quantify the findings of about 2,300 evaluations, and complement this data with our own survey of the evaluation authors. We show that the findings of evaluations are inconsistent with those of the academic literature on the output impacts of Cohesion Policy. Using further variation across Member States, our analysis suggests that the market of evaluations is rather oligopolistic within Member States, that it is very fragmented across the EU, and that there is often a strong involvement of managing authorities in the work of formally independent evaluators. These factors contribute to making the findings of the evaluations overly optimistic (beautiful) risking their relevance for (evidence-based) policy. We conclude by discussing reform options to make the evaluations of EU Cohesion Policy more unbiased and effective.

Suggested Citation

  • Zareh Asatryan & Carlo Birkholz & Friedrich Heinemann, 2025. "Evidence-based policy or beauty contest? An LLM-based meta-analysis of EU cohesion policy evaluations," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 32(2), pages 625-655, April.
  • Handle: RePEc:kap:itaxpf:v:32:y:2025:i:2:d:10.1007_s10797-024-09875-4
    DOI: 10.1007/s10797-024-09875-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10797-024-09875-4
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s10797-024-09875-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Philippe Aghion & Patrick Bolton & Christopher Harris & Bruno Jullien, 1991. "Optimal Learning by Experimentation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 58(4), pages 621-654.
    2. Shaoda Wang & David Y. Yang, 2021. "Policy Experimentation in China: the Political Economy of Policy Learning," NBER Working Papers 29402, National Bureau of Economic Research, Inc.
    3. Maximilian v. Ehrlich & Henry G. Overman, 2020. "Place-Based Policies and Spatial Disparities across European Cities," Journal of Economic Perspectives, American Economic Association, vol. 34(3), pages 128-149, Summer.
    4. Paul Resnick & Christopher Avery & Richard Zeckhauser, 1999. "The Market for Evaluations," American Economic Review, American Economic Association, vol. 89(3), pages 564-584, June.
    5. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    6. Hristos Doucouliagos & Martin Paldam, 2009. "The Aid Effectiveness Literature: The Sad Results Of 40 Years Of Research," Journal of Economic Surveys, Wiley Blackwell, vol. 23(3), pages 433-461, July.
    7. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    8. Heinemann, Friedrich & Asatryan, Zareh & Bachtrögler, Julia & Birkholz, Carlo & Corti, Franceso & von Ehrlich, Maximilian & Fratesi, Ugo & Fuest, Clemens & Lang, Valentin & Weber, Martin, 2024. "Enhancing objectivity and decision relevance: A better framework for evaluating cohesion policies," ZEW Discussion Papers 24-034, ZEW - Leibniz Centre for European Economic Research.
    9. Elliott Ash & Stephen Hansen, 2023. "Text Algorithms in Economics," Annual Review of Economics, Annual Reviews, vol. 15(1), pages 659-688, September.
    10. Zsolt Darvas & Jan Mazza & Catarina Midões, 2019. "How to improve European Union cohesion policy for the next decade," Bruegel Policy Contributions 30670, Bruegel.
    11. Hirsch, Alexander V., 2016. "Experimentation and Persuasion in Political Organizations," American Political Science Review, Cambridge University Press, vol. 110(1), pages 68-84, February.
    12. Zhaohui Wang, 2021. "The International Political Economy of China’s Exchange Rate Policy Making," Springer Books, Springer, number 978-981-33-4578-2, March.
    13. Asatryan, Zareh & Havlik, Annika & Heinemann, Friedrich & Nover, Justus, 2020. "Biases in fiscal multiplier estimates," European Journal of Political Economy, Elsevier, vol. 63(C).
    14. Steven Callander, 2011. "Searching and Learning by Trial and Error," American Economic Review, American Economic Association, vol. 101(6), pages 2277-2308, October.
    15. Paldam, Martin, 2018. "A model of the representative economist, as researcher and policy advisor," European Journal of Political Economy, Elsevier, vol. 54(C), pages 5-15.
    16. Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(1), pages 237-293.
    17. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org.
    18. Paolo Di Caro & Ugo Fratesi, 2022. "One policy, different effects: Estimating the region‐specific impacts of EU cohesion policy," Journal of Regional Science, Wiley Blackwell, vol. 62(1), pages 307-330, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Asatryan, Zareh & Birkholz, Carlo & Heinemann, Friedrich, 2024. "Evidence-based policy or beauty contest? An LLM-based meta-analysis of EU cohesion policy evaluations," ZEW Discussion Papers 24-037, ZEW - Leibniz Centre for European Economic Research.
    2. repec:osf:osfxxx:r3qng_v1 is not listed on IDEAS
    3. Samuel Chang & Andrew Kennedy & Aaron Leonard & John A. List, 2024. "12 Best Practices for Leveraging Generative AI in Experimental Research," NBER Working Papers 33025, National Bureau of Economic Research, Inc.
    4. Cova, Joshua & Schmitz, Luuk, 2024. "A primer for the use of classifier and generative large language models in social science research," OSF Preprints r3qng, Center for Open Science.
    5. Shumiao Ouyang & Hayong Yun & Xingjian Zheng, 2024. "How Ethical Should AI Be? How AI Alignment Shapes the Risk Preferences of LLMs," Papers 2406.01168, arXiv.org, revised Aug 2024.
    6. Kevin He & Ran Shorrer & Mengjia Xia, 2025. "Human Misperception of Generative-AI Alignment: A Laboratory Experiment," Papers 2502.14708, arXiv.org.
    7. Barrios, John & Lancieri, Filippo Maria & Levy, Joshua & Singh, Shashank & Valletti, Tommaso M. & Zingales, Luigi, 2024. "The conflict-of-interest discount in the marketplace of ideas," Working Papers 348, The University of Chicago Booth School of Business, George J. Stigler Center for the Study of the Economy and the State.
    8. Rosa-García, Alfonso, 2024. "Student Reactions to AI-Replicant Professor in an Econ101 Teaching Video," MPRA Paper 120135, University Library of Munich, Germany.
    9. Cujean, Julien & Bustamante, Maria Cecilia & Frésard, Laurent, 2019. "Knowledge Cycles and Corporate Investment," CEPR Discussion Papers 14152, C.E.P.R. Discussion Papers.
    10. Kevin Leyton-Brown & Paul Milgrom & Neil Newman & Ilya Segal, 2024. "Artificial Intelligence and Market Design: Lessons Learned from Radio Spectrum Reallocation," NBER Chapters, in: New Directions in Market Design, National Bureau of Economic Research, Inc.
    11. Capra, C. Monica & Kniesner, Thomas J., 2025. "Daniel Kahneman’s Underappreciated Last Published Paper: Empirical Implications for Benefit-Cost Analysis and a Chat Session Discussion with Bots," IZA Discussion Papers 17841, Institute of Labor Economics (IZA).
    12. Corina Cretu, 2024. "Cohesion Policy Agenda 2024-2029," Annals - Economy Series, Constantin Brancusi University, Faculty of Economics, vol. 5, pages 299-306, October.
    13. Kirshner, Samuel N., 2024. "GPT and CLT: The impact of ChatGPT's level of abstraction on consumer recommendations," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    14. Zengqing Wu & Run Peng & Xu Han & Shuyuan Zheng & Yixin Zhang & Chuan Xiao, 2023. "Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations," Papers 2311.06330, arXiv.org, revised Dec 2023.
    15. repec:osf:osfxxx:udz28_v1 is not listed on IDEAS
    16. Joshua C. Yang & Damian Dailisan & Marcin Korecki & Carina I. Hausladen & Dirk Helbing, 2024. "LLM Voting: Human Choices and AI Collective Decision Making," Papers 2402.01766, arXiv.org, revised Aug 2024.
    17. Nir Chemaya & Daniel Martin, 2023. "Perceptions and Detection of AI Use in Manuscript Preparation for Academic Journals," Papers 2311.14720, arXiv.org, revised Jan 2024.
    18. Lijia Ma & Xingchen Xu & Yong Tan, 2024. "Crafting Knowledge: Exploring the Creative Mechanisms of Chat-Based Search Engines," Papers 2402.19421, arXiv.org.
    19. Umberto Garfagnini & Bruno Strulovici, 2012. "Social Learning and Innovation Cycles (revision of DP#1516, The Dynamics of Innovation)," Discussion Papers 1546, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    20. Ali Goli & Amandeep Singh, 2023. "Exploring the Influence of Language on Time-Reward Perceptions in Large Language Models: A Study Using GPT-3.5," Papers 2305.02531, arXiv.org, revised Jun 2023.
    21. Evangelos Katsamakas, 2024. "Business models for the simulation hypothesis," Papers 2404.08991, arXiv.org.
    22. Daniel Aparicio-Pérez & Maria Teresa Balaguer-Coll & Emili Tortosa-Ausina, 2023. "On the relative contributions of national and regional institutions to economic development," Working Papers 2023/01, Economics Department, Universitat Jaume I, Castellón (Spain).

    More about this item

    Keywords

    Policy evaluation; EU cohesion policy; Large language model;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • H43 - Public Economics - - Publicly Provided Goods - - - Project Evaluation; Social Discount Rate
    • H54 - Public Economics - - National Government Expenditures and Related Policies - - - Infrastructures

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:itaxpf:v:32:y:2025:i:2:d:10.1007_s10797-024-09875-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.