IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0301301.html

The performance of interrupted time series designs with a limited number of time points: Learning losses due to school closures during the COVID-19 pandemic

Author

Listed:
  • Florian D van Leeuwen
  • Peter Lugtig
  • Remco Feskens

Abstract

Interrupted time series (ITS) designs are increasingly used for estimating the effect of shocks in natural experiments. Currently, ITS designs are often used in scenarios with many time points and simple data structures. This research investigates the performance of ITS designs when the number of time points is limited and with complex data structures. Using a Monte Carlo simulation study, we empirically derive the performance—in terms of power, bias and precision- of the ITS design. Scenarios are considered with multiple interventions, a low number of time points and different effect sizes based on a motivating example of the learning loss due to COVID school closures. The results of the simulation study show the power of the step change depends mostly on the sample size, while the power of the slope change depends on the number of time points. In the basic scenario, with both a step and a slope change and an effect size of 30% of the pre-intervention slope, the required sample size for detecting a step change is 1,100 with a minimum of twelve time points. For detecting a slope change the required sample size decreases to 500 with eight time points. To decide if there is enough power researchers should inspect their data, hypothesize about effect sizes and consider an appropriate model before applying an ITS design to their research. This paper contributes to the field of methodology in two ways. Firstly, the motivation example showcases the difficulty of employing ITS designs in cases which do not adhere to a single intervention. Secondly, models are proposed for more difficult ITS designs and their performance is tested.

Suggested Citation

  • Florian D van Leeuwen & Peter Lugtig & Remco Feskens, 2024. "The performance of interrupted time series designs with a limited number of time points: Learning losses due to school closures during the COVID-19 pandemic," PLOS ONE, Public Library of Science, vol. 19(8), pages 1-23, August.
    • Handle: RePEc:plo:pone00:0301301
    DOI: 10.1371/journal.pone.0301301
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0301301
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0301301&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0301301?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769, December.
    2. Ariel Linden, 2022. "Erratum: A comprehensive set of postestimation measures to enrich interrupted time-series analysis," Stata Journal, StataCorp LLC, vol. 22(1), pages 231-233, March.
    3. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    4. Clark, Andrew E. & Nong, Huifu & Zhu, Hongjia & Zhu, Rong, 2021. "Compensating for academic loss: Online learning and student performance during the COVID-19 pandemic," China Economic Review, Elsevier, vol. 68(C).
    5. Chou, Chih-Ping & Yang, Dongyun & Pentz, Mary Ann & Hser, Yih-Ing, 2004. "Piecewise growth curve modeling approach for longitudinal prevention study," Computational Statistics & Data Analysis, Elsevier, vol. 46(2), pages 213-225, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Junaid-ur-Rehman Siddiqui & Mansoor Ahmed Veesar & Kashif Manzoor & Irum Imran & Amir Saeed & Faisal Mahar & Saqib Ali Shaikh & Zafar Ali Dehraj & Aaliya Habib & Ghazunfer Abbas & Syed Azizur Rab & Vi, 2025. "Effects of the Challenge Initiative’s Community Health Volunteers (CHVs) on Public Sector Service Provision of Family Planning Services in Urban Sindh, Pakistan," IJERPH, MDPI, vol. 22(10), pages 1-14, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    2. Lutfah Ariana & Rimawan Pradiptyo & Evi Noor Afifah, 2026. "Assessing the impact of public–private partnership adoption on regional economic growth in Asia," International Economics and Economic Policy, Springer, vol. 23(2), pages 1-26, May.
    3. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, IZA Network @ LISER.
    4. Soonwoo Kwon & Liyang Sun, 2025. "Estimating Treatment Effects Under Bounded Heterogeneity," Papers 2510.05454, arXiv.org, revised Aug 2026.
    5. Shimizu, Hiroshi & Wakutsu, Naohiko, 2026. "Stimulating spin-outs and stalled subsequent technological development: Laser diodes in the United States and Japan under the SBIR program," Technological Forecasting and Social Change, Elsevier, vol. 223(C).
    6. Paul, Annemarie, 2015. "After work shopping? Employment effects of a deregulation of shop opening hours in the German retail sector," European Economic Review, Elsevier, vol. 80(C), pages 329-353.
    7. Imbens, Guido W., 2014. "Instrumental Variables: An Econometrician's Perspective," IZA Discussion Papers 8048, IZA Network @ LISER.
    8. Robert A. HartBy & Mirko Moro & J. Elizabeth Roberts, 2017. "Who gained from the introduction of free universal secondary education in England and Wales?," Oxford Economic Papers, Oxford University Press, vol. 69(3), pages 707-733.
    9. Antonis Adam & Sofia Tsarsitalidou, 2019. "Do sanctions lead to a decline in civil liberties?," Public Choice, Springer, vol. 180(3), pages 191-215, September.
    10. Wameq A. Raza & Ellen van de Poel & Arjun Bedi & Frans Rutten, 2016. "Impact of Community‐based Health Insurance on Access and Financial Protection: Evidence from Three Randomized Control Trials in Rural India," Health Economics, John Wiley & Sons, Ltd., vol. 25(6), pages 675-687, June.
    11. Alexei Roudnitski & Somwrita Sarkar, 2026. "The effect of policy regulations in the short-term rental platform market on long-term rental prices: A case study of Airbnb in Sydney," Environment and Planning B, , vol. 53(4), pages 932-955, May.
    12. repec:dgr:rugsom:14009-eef is not listed on IDEAS
    13. Moshoeshoe,Ramaele Elias, 2020. "Long-Term Effects of Free Primary Education on Educational Achievement : Evidence from Lesotho," Policy Research Working Paper Series 9404, The World Bank.
    14. Martin Huber, 2024. "An introduction to causal discovery," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 160(1), pages 1-16, December.
    15. Antonis Adam & Sofia Tsarsitalidou, 2023. "Serving two masters: the effect of state religion on fiscal capacity," Public Choice, Springer, vol. 194(1), pages 181-203, January.
    16. Daniel Morales Martínez & Alexandre Gori Maia, 2018. "The Impacts of Cash Transfers on Subjective Wellbeing and Poverty: The Case of Colombia," Journal of Family and Economic Issues, Springer, vol. 39(4), pages 616-633, December.
    17. Ni, Xiaoran & Song, Wei & Yao, Jiaquan, 2020. "Stakeholder orientation and corporate payout policy: Insights from state legal shocks," Journal of Banking & Finance, Elsevier, vol. 121(C).
    18. Kadir Atalay & Garry F. Barrett & Anita Staneva, 2020. "The effect of retirement on home production: evidence from Australia," Review of Economics of the Household, Springer, vol. 18(1), pages 117-139, March.
    19. Diop, Thierno Bocar & Védrine, Lionel, 2025. "Did crop diversity criterion from CAP green payments affect both economic and environmental farm performances? Quasi-experimental evidence from France," Ecological Economics, Elsevier, vol. 227(C).
    20. Paul Goldsmith-Pinkham & Peter Hull & Michal Kolesár, 2024. "Contamination Bias in Linear Regressions," American Economic Review, American Economic Association, vol. 114(12), pages 4015-4051, December.
    21. Andor, Mark A. & Fels, Katja M., 2018. "Behavioral Economics and Energy Conservation – A Systematic Review of Non-price Interventions and Their Causal Effects," Ecological Economics, Elsevier, vol. 148(C), pages 178-210.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0301301. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.