IDEAS home Printed from https://ideas.repec.org/a/kap/policy/v58y2025i1d10.1007_s11077-024-09564-3.html
   My bibliography  Save this article

Assessing evidence based on scale can be a useful predictor of policy outcomes

Author

Listed:
  • Kai Ruggeri

    (Columbia University)

Abstract

With growing interest in more formalized applications of scientific evidence to policy, there are concerns about what evidence is selected and applied, and for what purpose. We present an initial argument that scale of evidence could be used in policy decisions in ways that can usefully predict effectiveness of policy interventions. This is valuable given that, as we show using a survey of of 251 policymakers, there is no single type of evidence (e.g., RCTs, systematic reviews, surveys) that is "best" to all policymakers or all policy domains. By simply rating the "level" of studies' size and scope used to inform policies, we show how high levels of evidence were more strongly associated with better (i.e., intended) outcomes across 82 policies. The rate of policies achieving intended outcomes ranged from 38%, when no evidence was available prior to the policy, to 78%, when large-scale evidence existed prior to implementation. Though these findings are encouraging, this piece is largely meant to argue for, not universally validate, a simple approach to assess evidence appropriately when making policy decisions. Instead, we argue that using this approach in combination with other ratings may better serve applications of evidence to achieve better outcomes for populations.

Suggested Citation

  • Kai Ruggeri, 2025. "Assessing evidence based on scale can be a useful predictor of policy outcomes," Policy Sciences, Springer;Society of Policy Sciences, vol. 58(1), pages 179-188, March.
  • Handle: RePEc:kap:policy:v:58:y:2025:i:1:d:10.1007_s11077-024-09564-3
    DOI: 10.1007/s11077-024-09564-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11077-024-09564-3
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s11077-024-09564-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kai Ruggeri & Friederike Stock & S. Alexander Haslam & Valerio Capraro & Paulo Boggio & Naomi Ellemers & Aleksandra Cichocka & Karen M. Douglas & David G. Rand & Sander Linden & Mina Cikara & Eli J. F, 2024. "A synthesis of evidence for policy from behavioural science during COVID-19," Nature, Nature, vol. 625(7993), pages 134-147, January.
    2. Marco Liverani & Benjamin Hawkins & Justin O Parkhurst, 2013. "Political and Institutional Influences on the Use of Evidence in Public Health Policy. A Systematic Review," PLOS ONE, Public Library of Science, vol. 8(10), pages 1-9, October.
    3. Qianjin Zong & Zhihong Huang & Jiaru Huang, 2023. "Do open science badges work? Estimating the effects of open science badges on an article’s social media attention and research impacts," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3627-3648, June.
    4. Moshe Maor, 2020. "Policy over- and under-design: an information quality perspective," Policy Sciences, Springer;Society of Policy Sciences, vol. 53(3), pages 395-411, September.
    5. Lee, Nathan, 2022. "Do Policy Makers Listen to Experts? Evidence from a National Survey of Local and State Policy Makers," American Political Science Review, Cambridge University Press, vol. 116(2), pages 677-688, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Steve R Makkar & Abby Haynes & Anna Williamson & Sally Redman, 2018. "Organisational capacity and its relationship to research use in six Australian health policy agencies," PLOS ONE, Public Library of Science, vol. 13(3), pages 1-22, March.
    2. Kaba, Mustafa & Koyuncu, Murat & Schneider, Sebastian O. & Sutter, Matthias, 2024. "Social norms, political polarization, and vaccination attitudes: Evidence from a survey experiment in Turkey," European Economic Review, Elsevier, vol. 168(C).
    3. Lukas Breide & Oliver Budzinski & Thomas Grebel & Juliane Mendelsohn, 2025. "Forerunners vs. latecomers—institutional competition in the German federalism during the COVID crisis," European Journal of Law and Economics, Springer, vol. 59(1), pages 101-132, February.
    4. Anders Kärnä & Patrik Öhberg, 2023. "Misrepresentation and migration," Kyklos, Wiley Blackwell, vol. 76(4), pages 503-525, November.
    5. Béné, Christophe, 2022. "Why the Great Food Transformation may not happen – A deep-dive into our food systems’ political economy, controversies and politics of evidence," World Development, Elsevier, vol. 154(C).
    6. Cyr, Pascale Renée & Jain, Vageesh & Chalkidou, Kalipso & Ottersen, Trygve & Gopinathan, Unni, 2021. "Evaluations of public health interventions produced by health technology assessment agencies: A mapping review and analysis by type and evidence content," Health Policy, Elsevier, vol. 125(8), pages 1054-1064.
    7. Ryan Petteway & Mahasin Mujahid & Amani Allen & Rachel Morello-Frosch, 2019. "Towards a People’s Social Epidemiology: Envisioning a More Inclusive and Equitable Future for Social Epi Research and Practice in the 21st Century," IJERPH, MDPI, vol. 16(20), pages 1-21, October.
    8. Saville, Christopher W.N. & Mann, Robin & Lockard, Anthony Scott & Bark-Connell, Aidan & Gabuljah, Stella Gmekpebi & Young, April M. & Thomas, Daniel Rhys, 2024. "Covid and the common good: In-group out-group dynamics and Covid-19 vaccination in Wales and the United States," Social Science & Medicine, Elsevier, vol. 352(C).
    9. Pol Campos-Mercade & Armando N. Meier & Stephan Meier & Devin Pope & Florian H. Schneider & Erik Wengstroem, 2025. "Incentives to Vaccinate," CEBI working paper series 24-15, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    10. Weldon, Isaac & Parkhurst, Justin, 2022. "Governing evidence use in the nutrition policy process: evidence and lessons from the 2020 Canada food guide," LSE Research Online Documents on Economics 112430, London School of Economics and Political Science, LSE Library.
    11. Kyung-Eun (Anna) Choi & Lara Lindert & Lara Schlomann & Holger Pfaff, 2022. "“I’ll leave that to the case managers.” Healthcare Service Providers‘ Perceptions of Organizational Readiness for Change in a Randomized Controlled Trial—A Qualitative Analysis Exploring Implementatio," IJERPH, MDPI, vol. 19(9), pages 1-17, May.
    12. Wei Yu & Junpeng Chen & Sanhong Deng, 2024. "Open Science Under Debate: Disentangling the Interest on Twitter and Scholarly Research," SAGE Open, , vol. 14(3), pages 21582440241, August.
    13. Edwards, Rachael C. & Kneale, Dylan & Stansfield, Claire & Lester, Sarah, 2024. "What are the mechanisms driving the early stages of embedded researcher interventions? A qualitative process evaluation in English local government," Social Science & Medicine, Elsevier, vol. 340(C).
    14. Kathryn Oliver & Warren Pearce, 2017. "Three lessons from evidence-based medicine and policy: increase transparency, balance inputs and understand power," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-7, December.
    15. Kadidiatou Kadio & Christian Dagenais & Valery Ridde, 2023. "How does explicit knowledge inform policy shaping? The case of Burkina Faso’s national social protection policy," PLOS ONE, Public Library of Science, vol. 18(4), pages 1-22, April.
    16. Pol Campos-Mercade & Armando N. Meier & Stephan Meier & Devin G. Pope & Florian H. Schneider & Erik Wengström, 2024. "Incentives to Vaccinate," NBER Working Papers 32899, National Bureau of Economic Research, Inc.
    17. P Th Houngbo & H L S Coleman & M Zweekhorst & Tj De Cock Buning & D Medenou & J F G Bunders, 2017. "A Model for Good Governance of Healthcare Technology Management in the Public Sector: Learning from Evidence-Informed Policy Development and Implementation in Benin," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-22, January.
    18. Choi, Dahyun, 2025. "How Politics Percolates Through Science Assessment," OSF Preprints ujyec, Center for Open Science.
    19. Zoltán Kaló & Adrian Gheorghe & Mirjana Huic & Marcell Csanádi & Finn Boerlum Kristensen, 2016. "HTA Implementation Roadmap in Central and Eastern European Countries," Health Economics, John Wiley & Sons, Ltd., vol. 25(S1), pages 179-192, February.
    20. Lee, Chien-Chiang & Zou, Jinyang, 2024. "Impacts of the energy transition on public health in the context of country risk: From an international perspective," Economic Analysis and Policy, Elsevier, vol. 83(C), pages 873-895.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:policy:v:58:y:2025:i:1:d:10.1007_s11077-024-09564-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.