IDEAS home Printed from https://ideas.repec.org/a/eee/hepoli/v104y2012i3p206-214.html
   My bibliography  Save this article

Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: A meta-review

Author

Listed:
  • Rockers, Peter C.
  • Feigl, Andrea B.
  • Røttingen, John-Arne
  • Fretheim, Atle
  • de Ferranti, David
  • Lavis, John N.
  • Melberg, Hans Olav
  • Bärnighausen, Till

Abstract

At present, there exists no widely agreed upon set of study-design selection criteria for systematic reviews of health systems research, except for those proposed by the Cochrane Collaboration's Effective Practice and Organisation of Care (EPOC) review group (which comprises randomized controlled trials, controlled clinical trials, controlled before-after studies, and interrupted time series). We conducted a meta-review of the study-design selection criteria used in systematic reviews available in the McMaster University's Health Systems Evidence or the EPOC database. Of 414 systematic reviews, 13% did not indicate any study-design selection criteria. Of the 359 studies that described such criteria, 50% limited their synthesis to controlled trials and 68% to some or all of the designs defined by the EPOC criteria. Seven out of eight reviews identified at least one controlled trial that was relevant for the review topic. Seven percent of the reviews included either no or only one relevant primary study. Our meta-review reveals reviewers’ preferences for restricting synthesis to controlled experiments or study designs that comply with the EPOC criteria. We discuss the advantages and disadvantages of the current practices regarding study-design selection in systematic reviews of health systems research as well as alternative approaches.

Suggested Citation

  • Rockers, Peter C. & Feigl, Andrea B. & Røttingen, John-Arne & Fretheim, Atle & de Ferranti, David & Lavis, John N. & Melberg, Hans Olav & Bärnighausen, Till, 2012. "Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: A meta-review," Health Policy, Elsevier, vol. 104(3), pages 206-214.
    • Handle: RePEc:eee:hepoli:v:104:y:2012:i:3:p:206-214
    DOI: 10.1016/j.healthpol.2011.12.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0168851011002740
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.healthpol.2011.12.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    2. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    3. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    4. David Card & Carlos Dobkin & Nicole Maestas, 2008. "The Impact of Nearly Universal Insurance Coverage on Health Care Utilization: Evidence from Medicare," American Economic Review, American Economic Association, vol. 98(5), pages 2242-2258, December.
    5. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    6. Bärnighausen, Till & Sauerborn, Rainer, 2002. "One hundred and eighteen years of the German health insurance system: are there any lessons for middle- and low-income countries?," Social Science & Medicine, Elsevier, vol. 54(10), pages 1559-1587, May.
    7. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009 is not listed on IDEAS
    8. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009.pdf is not listed on IDEAS
    9. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    10. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    11. repec:pri:rpdevs:instruments_of_development.pdf is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rockers, Peter C. & Røttingen, John-Arne & Shemilt, Ian & Tugwell, Peter & Bärnighausen, Till, 2015. "Inclusion of quasi-experimental studies in systematic reviews of health systems research," Health Policy, Elsevier, vol. 119(4), pages 511-521.
    2. Nga Le Thi Quynh & Groot, Wim & Tomini, Sonila M. & Tomini, Florian, 2017. "Effects of health insurance on labour supply: A systematic review," MERIT Working Papers 2017-017, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    3. Xiu-xia, Li & Ya, Zheng & Yao-long, Chen & Ke-hu, Yang & Zong-jiu, Zhang, 2015. "The reporting characteristics and methodological quality of Cochrane reviews about health policy research," Health Policy, Elsevier, vol. 119(4), pages 503-510.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. German Blanco & Carlos A. Flores & Alfonso Flores-Lagunes, 2013. "Bounds on Average and Quantile Treatment Effects of Job Corps Training on Wages," Journal of Human Resources, University of Wisconsin Press, vol. 48(3), pages 659-701.
    2. Md. Jahir Uddin Palas & Fernando Moreira, 2022. "The impact of government assistance on banks' efficiency," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 27(2), pages 1731-1748, April.
    3. repec:dgr:rugsom:14009-eef is not listed on IDEAS
    4. Öberg, Stefan, 2021. "Treatment for natural experiments: How to improve causal estimates using conceptual definitions and substantive interpretations," SocArXiv pkyue, Center for Open Science.
    5. Jérémie Gignoux, 2006. "Évaluations ex ante et ex post d'un programme d'allocations scolaires conditionnées au Mexique," Economie & Prévision, La Documentation Française, vol. 0(3), pages 59-85.
    6. Susanne Prantl & Frederik Thenée, 2017. "Incorporation Decisions and Job Creation in New Firms," Working Paper Series in Economics 91, University of Cologne, Department of Economics.
    7. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    8. Fauver, Larry & Hung, Mingyi & Li, Xi & Taboada, Alvaro G., 2017. "Board reforms and firm value: Worldwide evidence," Journal of Financial Economics, Elsevier, vol. 125(1), pages 120-142.
    9. Kugler Franziska & Schwerdt Guido & Wößmann Ludger, 2014. "Ökonometrische Methoden zur Evaluierung kausaler Effekte der Wirtschaftspolitik," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 15(2), pages 105-132, June.
    10. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    11. Parrotta, Pierpaolo & Pozzoli, Dario & Sala, Davide, 2016. "Ethnic diversity and firms' export behavior," European Economic Review, Elsevier, vol. 89(C), pages 248-263.
    12. Foster, E. Michael & McCombs-Thornton, Kimberly, 2013. "Child welfare and the challenge of causal inference," Children and Youth Services Review, Elsevier, vol. 35(7), pages 1130-1142.
    13. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    14. Yingying Dong & Arthur Lewbel, 2011. "Regression Discontinuity Marginal Threshold Treatment Effects," Working Papers 111205, University of California-Irvine, Department of Economics.
    15. Masakure, Oliver, 2016. "The effect of employee loyalty on wages," Journal of Economic Psychology, Elsevier, vol. 56(C), pages 274-298.
    16. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    17. Dana Rotz & Brian Goesling & Nicholas Redel & Menbere Shiferaw & Claire Smither-Wulsin, "undated". "Assessing the Benefits of Delayed Sexual Activity: A Synthesis of the Literature," Mathematica Policy Research Reports 16a96ac7ac69493eaeb7edba2, Mathematica Policy Research.
    18. Bonin, Holger & Rinne, Ulf, 2017. "Machbarkeitsstudie zur Durchführung einer Evaluation der arbeitsmarktpolitischen Integrationsmaßnahmen für Flüchtlinge," IZA Research Reports 76, Institute of Labor Economics (IZA).
    19. Simphiwe Innocentia Hlatshwayo & Rob Slotow & Mjabuliseni Simon Cloapas Ngidi, 2023. "The Role of Smallholder Farming on Rural Household Dietary Diversity," Agriculture, MDPI, vol. 13(3), pages 1-16, February.
    20. Faruque As Sunny & Linlin Fu & Md Sadique Rahman & Zuhui Huang, 2022. "Determinants and Impact of Solar Irrigation Facility (SIF) Adoption: A Case Study in Northern Bangladesh," Energies, MDPI, vol. 15(7), pages 1-17, March.
    21. Adewale M. Ogunmodede & Mary O. Ogunsanwo & Victor Manyong, 2020. "Unlocking the Potential of Agribusiness in Africa through Youth Participation: An Impact Evaluation of N-Power Agro Empowerment Program in Nigeria," Sustainability, MDPI, vol. 12(14), pages 1-18, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:hepoli:v:104:y:2012:i:3:p:206-214. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu or the person in charge (email available below). General contact details of provider: http://www.elsevier.com/locate/healthpol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.