IDEAS home Printed from https://ideas.repec.org/a/eee/hepoli/v119y2015i4p511-521.html
   My bibliography  Save this article

Inclusion of quasi-experimental studies in systematic reviews of health systems research

Author

Listed:
  • Rockers, Peter C.
  • Røttingen, John-Arne
  • Shemilt, Ian
  • Tugwell, Peter
  • Bärnighausen, Till

Abstract

Systematic reviews of health systems research commonly limit studies for evidence synthesis to randomized controlled trials. However, well-conducted quasi-experimental studies can provide strong evidence for causal inference. With this article, we aim to stimulate and inform discussions on including quasi-experiments in systematic reviews of health systems research. We define quasi-experimental studies as those that estimate causal effect sizes using exogenous variation in the exposure of interest that is not directly controlled by the researcher. We incorporate this definition into a non-hierarchical three-class taxonomy of study designs – experiments, quasi-experiments, and non-experiments. Based on a review of practice in three disciplines related to health systems research (epidemiology, economics, and political science), we discuss five commonly used study designs that fit our definition of quasi-experiments: natural experiments, instrumental variable analyses, regression discontinuity analyses, interrupted times series studies, and difference studies including controlled before-and-after designs, difference-in-difference designs and fixed effects analyses of panel data. We further review current practices regarding quasi-experimental studies in three non-health fields that utilize systematic reviews (education, development, and environment studies) to inform the design of approaches for synthesizing quasi-experimental evidence in health systems research. Ultimately, the aim of any review is practical: to provide useful information for policymakers, practitioners, and researchers. Future work should focus on building a consensus among users and producers of systematic reviews regarding the inclusion of quasi-experiments.

Suggested Citation

  • Rockers, Peter C. & Røttingen, John-Arne & Shemilt, Ian & Tugwell, Peter & Bärnighausen, Till, 2015. "Inclusion of quasi-experimental studies in systematic reviews of health systems research," Health Policy, Elsevier, vol. 119(4), pages 511-521.
    • Handle: RePEc:eee:hepoli:v:119:y:2015:i:4:p:511-521
    DOI: 10.1016/j.healthpol.2014.10.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S016885101400270X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.healthpol.2014.10.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    2. Douglas Almond & Joseph J. Doyle & Amanda E. Kowalski & Heidi Williams, 2010. "Estimating Marginal Returns to Medical Care: Evidence from At-risk Newborns," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(2), pages 591-634.
    3. Amy Finkelstein & Sarah Taubman & Bill Wright & Mira Bernstein & Jonathan Gruber & Joseph P. Newhouse & Heidi Allen & Katherine Baicker, 2012. "The Oregon Health Insurance Experiment: Evidence from the First Year," The Quarterly Journal of Economics, Oxford University Press, vol. 127(3), pages 1057-1106.
    4. Hugh Waddington & Howard White & Birte Snilstveit & Jorge Garcia Hombrados & Martina Vojtkova & Philip Davies & Ami Bhavsar & John Eyers & Tracey Perez Koehlmoos & Mark Petticrew & Jeffrey C. Valentin, 2012. "How to do a good systematic review of effects in international development: a tool kit," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 4(3), pages 359-387, September.
    5. Robinson, Gregory & McNulty, John E. & Krasno, Jonathan S., 2009. "Observing the Counterfactual? The Search for Political Experiments in Nature," Political Analysis, Cambridge University Press, vol. 17(4), pages 341-357.
    6. William Shadish David Myers, 2004. "Campbell Collaboration Research Design Policy Brief," Mathematica Policy Research Reports 05f02566dfa14264954aedc6b, Mathematica Policy Research.
    7. Stephen G. Donald & Kevin Lang, 2007. "Inference with Difference-in-Differences and Other Panel Data," The Review of Economics and Statistics, MIT Press, vol. 89(2), pages 221-233, May.
    8. Su Golder & Yoon K Loke & Martin Bland, 2011. "Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview," PLOS Medicine, Public Library of Science, vol. 8(5), pages 1-13, May.
    9. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    10. Meyer, Bruce D, 1995. "Natural and Quasi-experiments in Economics," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(2), pages 151-161, April.
    11. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    12. repec:mpr:mprres:4684 is not listed on IDEAS
    13. King, Gary & Keohane, Robert O. & Verba, Sidney, 1995. "The Importance of Research Design in Political Science," American Political Science Review, Cambridge University Press, vol. 89(2), pages 475-481, June.
    14. Rockers, Peter C. & Feigl, Andrea B. & Røttingen, John-Arne & Fretheim, Atle & de Ferranti, David & Lavis, John N. & Melberg, Hans Olav & Bärnighausen, Till, 2012. "Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: A meta-review," Health Policy, Elsevier, vol. 104(3), pages 206-214.
    15. Paul L Hutchinson & Dominique Meekers, 2012. "Estimating Causal Effects from Family Planning Health Communication Campaigns Using Panel Data: The “Your Health, Your Wealth” Campaign in Egypt," PLOS ONE, Public Library of Science, vol. 7(9), pages 1-14, September.
    16. Hahn, Jinyong & Todd, Petra & Van der Klaauw, Wilbert, 2001. "Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design," Econometrica, Econometric Society, vol. 69(1), pages 201-209, January.
    17. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    18. Sara Bennett & Irene Akua Agyepong & Kabir Sheikh & Kara Hanson & Freddie Ssengooba & Lucy Gilson, 2011. "Building the Field of Health Policy and Systems Research: An Agenda for Action," PLOS Medicine, Public Library of Science, vol. 8(8), pages 1-5, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Eric L. Sevigny & Rosalie L. Pacula & Ariel M. Aloe & Danye N. Medhin & Jared Greathouse, 2021. "PROTOCOL: The effects of cannabis liberalization laws on health, safety, and socioeconomic outcomes: An evidence and gap map," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(1), March.
    2. Stanley, T. D. & Doucouliagos, Hristos & Havranek, Tomas, 2023. "Meta-analyses of partial correlations are biased: Detection and solutions," EconStor Preprints 270940, ZBW - Leibniz Information Centre for Economics.
    3. Nga Le Thi Quynh & Groot, Wim & Tomini, Sonila M. & Tomini, Florian, 2017. "Effects of health insurance on labour supply: A systematic review," MERIT Working Papers 2017-017, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    4. Susan P Sparkes & Rifat Atun & Till Bӓrnighausen, 2019. "The impact of the Family Medicine Model on patient satisfaction in Turkey: Panel analysis with province fixed effects," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-13, January.
    5. Patrik Grahn & Anna María Pálsdóttir & Johan Ottosson & Ingibjörg H. Jonsdottir, 2017. "Longer Nature-Based Rehabilitation May Contribute to a Faster Return to Work in Patients with Reactions to Severe Stress and/or Depression," IJERPH, MDPI, vol. 14(11), pages 1-17, October.
    6. Hanchao Cheng & Yuou Zhang & Jing Sun & Yuanli Liu, 2022. "How Did Zero-Markup Medicines Policy Change Prescriptions in the Eyes of Patients?—A Retrospective Quasi-Experimental Analysis," IJERPH, MDPI, vol. 19(19), pages 1-11, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    2. Cuong Viet Nguyen & Finn Tarp, 2023. "Cash Transfers and Labor Supply: New Evidence on Impacts and Mechanisms," DERG working paper series 23-18, University of Copenhagen. Department of Economics. Development Economics Research Group (DERG).
    3. Michael Lechner, 2002. "Mikroökonometrische Evaluation arbeitsmarktpolitischer Massnahmen," University of St. Gallen Department of Economics working paper series 2002 2002-20, Department of Economics, University of St. Gallen.
    4. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    5. Timothy F. Harris & Aaron Yelowitz, 2018. "Life Insurance Holdings And Well‐Being Of Surviving Spouses," Contemporary Economic Policy, Western Economic Association International, vol. 36(3), pages 526-538, July.
    6. Shin-Yi Chou & Jin-Tan Liu & James Hammitt, 2006. "Households’ precautionary behaviors—the effects of the introduction of National Health Insurance in Taiwan," Review of Economics of the Household, Springer, vol. 4(4), pages 395-421, December.
    7. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    8. Feng, Andy & Graetz, Georg, 2017. "A question of degree: The effects of degree class on labor market outcomes," Economics of Education Review, Elsevier, vol. 61(C), pages 140-161.
    9. Timothy J. Bartik, 2004. "Evaluating the Impacts of Local Economic Development Policies on Local Economic Outcomes: What Has Been Done and What Is Doable?," Book chapters authored by Upjohn Institute researchers, in: Evaluating Local Economic and Employment Development: How to Access Waht Works Among Programmes and Policies, pages 113-142, W.E. Upjohn Institute for Employment Research.
    10. Mazzutti, Caio Cícero Toledo Piza da Costa, 2016. "Three essays on the causal impacts of child labour laws in Brazil," Economics PhD Theses 0616, Department of Economics, University of Sussex Business School.
    11. Nicolas R. Ziebarth, 2018. "Social Insurance and Health," Contributions to Economic Analysis, in: Health Econometrics, volume 127, pages 57-84, Emerald Group Publishing Limited.
    12. Christopher F. Parmeter & Jaren C. Pope, 2013. "Quasi-experiments and hedonic property value methods," Chapters, in: John A. List & Michael K. Price (ed.), Handbook on Experimental Economics and the Environment, chapter 1, pages 3-66, Edward Elgar Publishing.
    13. Arnaud Cedric Kamkoum, 2023. "The Federal Reserve's Response to the Global Financial Crisis and Its Long-Term Impact: An Interrupted Time-Series Natural Experimental Analysis," Papers 2305.12318, arXiv.org.
    14. Jin-young Choi & Myoung-jae Lee, 2017. "Regression discontinuity: review with extensions," Statistical Papers, Springer, vol. 58(4), pages 1217-1246, December.
    15. Tenn, Steven & Herman, Douglas A. & Wendling, Brett, 2010. "The role of education in the production of health: An empirical analysis of smoking behavior," Journal of Health Economics, Elsevier, vol. 29(3), pages 404-417, May.
    16. Martin-Anderson, Sarah, 2013. "Prenatal attitudes and parity predict selection into a U.S. child health program: A short report," Social Science & Medicine, Elsevier, vol. 95(C), pages 128-132.
    17. Foster, E. Michael & McCombs-Thornton, Kimberly, 2013. "Child welfare and the challenge of causal inference," Children and Youth Services Review, Elsevier, vol. 35(7), pages 1130-1142.
    18. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.
    19. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    20. Bernal, Noelia & Carpio, Miguel A. & Klein, Tobias J., 2017. "The effects of access to health insurance: Evidence from a regression discontinuity design in Peru," Journal of Public Economics, Elsevier, vol. 154(C), pages 122-136.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:hepoli:v:119:y:2015:i:4:p:511-521. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu or the person in charge (email available below). General contact details of provider: http://www.elsevier.com/locate/healthpol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.