IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0301301.html
   My bibliography  Save this article

The performance of interrupted time series designs with a limited number of time points: Learning losses due to school closures during the COVID-19 pandemic

Author

Listed:
  • Florian D van Leeuwen
  • Peter Lugtig
  • Remco Feskens

Abstract

Interrupted time series (ITS) designs are increasingly used for estimating the effect of shocks in natural experiments. Currently, ITS designs are often used in scenarios with many time points and simple data structures. This research investigates the performance of ITS designs when the number of time points is limited and with complex data structures. Using a Monte Carlo simulation study, we empirically derive the performance—in terms of power, bias and precision- of the ITS design. Scenarios are considered with multiple interventions, a low number of time points and different effect sizes based on a motivating example of the learning loss due to COVID school closures. The results of the simulation study show the power of the step change depends mostly on the sample size, while the power of the slope change depends on the number of time points. In the basic scenario, with both a step and a slope change and an effect size of 30% of the pre-intervention slope, the required sample size for detecting a step change is 1,100 with a minimum of twelve time points. For detecting a slope change the required sample size decreases to 500 with eight time points. To decide if there is enough power researchers should inspect their data, hypothesize about effect sizes and consider an appropriate model before applying an ITS design to their research. This paper contributes to the field of methodology in two ways. Firstly, the motivation example showcases the difficulty of employing ITS designs in cases which do not adhere to a single intervention. Secondly, models are proposed for more difficult ITS designs and their performance is tested.

Suggested Citation

  • Florian D van Leeuwen & Peter Lugtig & Remco Feskens, 2024. "The performance of interrupted time series designs with a limited number of time points: Learning losses due to school closures during the COVID-19 pandemic," PLOS ONE, Public Library of Science, vol. 19(8), pages 1-23, August.
    • Handle: RePEc:plo:pone00:0301301
    DOI: 10.1371/journal.pone.0301301
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0301301
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0301301&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0301301?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Ariel Linden, 2017. "A comprehensive set of postestimation measures to enrich interrupted time-series analysis," Stata Journal, StataCorp LLC, vol. 17(1), pages 73-88, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ellison, Richard B. & Ellison, Adrian B. & Greaves, Stephen P. & Sampaio, Breno, 2017. "Electronic ticketing systems as a mechanism for travel behaviour change? Evidence from Sydney’s Opal card," Transportation Research Part A: Policy and Practice, Elsevier, vol. 99(C), pages 80-93.
    2. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    3. Valeria Bejarano-Salcedo & William Iván Moreno-Jimenez & Juan Manuel Julio-Román, 2020. "La Magnitud y Duración del Efecto de la Intervención por Subastas sobre el Mercado Cambiario: El caso Colombiano," Borradores de Economia 1142, Banco de la Republica de Colombia.
    4. Dennis Shen & Peng Ding & Jasjeet Sekhon & Bin Yu, 2022. "Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data," Papers 2207.14481, arXiv.org, revised Oct 2022.
    5. Caballero, Julián, 2021. "Corporate dollar debt and depreciations: All’s well that ends well?," Journal of Banking & Finance, Elsevier, vol. 130(C).
    6. Tymon Słoczyński, 2022. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," The Review of Economics and Statistics, MIT Press, vol. 104(3), pages 501-509, May.
    7. Robert A. HartBy & Mirko Moro & J. Elizabeth Roberts, 2017. "Who gained from the introduction of free universal secondary education in England and Wales?," Oxford Economic Papers, Oxford University Press, vol. 69(3), pages 707-733.
    8. Peter Backus & Thien Nguyen, 2021. "The Effect of the Sex Buyer Law on the Market for Sex, Sexual Health and Sexual Violence," Economics Discussion Paper Series 2106, Economics, The University of Manchester.
    9. R. Forrest McCluer & Martha A. Starr, 2013. "Using Difference in Differences to Estimate Damages in Healthcare Antitrust: A Case Study of Marshfield Clinic," International Journal of the Economics of Business, Taylor & Francis Journals, vol. 20(3), pages 447-469, November.
    10. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    11. Luís Felipe Batista de Oliveira & Rafael Terra, 2016. "Impact of school day extension on educational outcomes: evidence from Mais Educação in Brazil," One Pager 329, International Policy Centre for Inclusive Growth.
    12. Cammeraat, Emile & Jongen, Egbert L. W. & Koning, Pierre, 2017. "Preventing NEETs during the Great Recession: The Effects of a Mandatory Activation Program for Young Welfare Recipients," IZA Discussion Papers 11090, Institute of Labor Economics (IZA).
    13. Jones, Kelly W. & Muñoz Brenes, Carlos L. & Shinbrot, Xoco A. & López-Báez, Walter & Rivera-Castañeda, Andrómeda, 2018. "The influence of cash and technical assistance on household-level outcomes in payments for hydrological services programs in Chiapas, Mexico," Ecosystem Services, Elsevier, vol. 31(PA), pages 208-218.
    14. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    15. Rainer Eppel & Marian Fink & Helmut Mahringer, 2016. "Die Wirkung zentraler Interventionen des AMS im Prozess der Vermittlung von Arbeitslosen," WIFO Studies, WIFO, number 59029.
    16. Boockmann Bernhard & Buch Claudia M. & Schnitzer Monika, 2014. "Evidenzbasierte Wirtschaftspolitik in Deutschland: Defizite und Potentiale," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 15(4), pages 307-323, December.
    17. Ljungqvist, Alexander & Farre-Mensa, Joan & Hegde, Deepak, 2016. "The Bright Side of Patents," CEPR Discussion Papers 11091, C.E.P.R. Discussion Papers.
    18. Sara Wong, 2017. "Minimum wage impacts on wages and hours worked of low-income workers in Ecuador," Working Papers PMMA 2017-14, PEP-PMMA.
    19. Maldonado, Stanislao, 2014. "The Non-Monotonic Political Effects of Resource Booms," MPRA Paper 85649, University Library of Munich, Germany, revised 18 Dec 2017.
    20. Ferreira, Maria & de Grip, Andries & van der Velden, Rolf, 2018. "Does informal learning at work differ between temporary and permanent workers? Evidence from 20 OECD countries," Labour Economics, Elsevier, vol. 55(C), pages 18-40.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0301301. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.