IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1911.03764.html
   My bibliography  Save this paper

Optimal Experimental Design for Staggered Rollouts

Author

Listed:
  • Ruoxuan Xiong
  • Susan Athey
  • Mohsen Bayati
  • Guido Imbens

Abstract

Experimentation has become an increasingly prevalent tool for guiding policy choices, firm decisions, and product innovation. A common hurdle in designing experiments is the lack of statistical power. In this paper, we study optimal multi-period experimental design under the constraint that the treatment cannot be easily removed once implemented; for example, a government or firm might implement treatment in different geographies at different times, where the treatment cannot be easily removed due to practical constraints. The design problem is to select which units to treat at which time, intending to test hypotheses about the effect of the treatment. When the potential outcome is a linear function of a unit effect, a time effect, and observed discrete covariates, we provide an analytically feasible solution to the design problem where the variance of the estimator for the treatment effect is at most 1+O(1/N^2) times the variance of the optimal design, where N is the number of units. This solution assigns units in a staggered treatment adoption pattern, where the proportion treated is a linear function of time. In the general setting where outcomes depend on latent covariates, we show that historical data can be utilized in the optimal design. We propose a data-driven local search algorithm with the minimax decision criterion to assign units to treatment times. We demonstrate that our approach improves upon benchmark experimental designs through synthetic experiments on real-world data sets from several domains, including healthcare, finance, and retail. Finally, we consider the case where the treatment effect changes with the time of treatment, showing that the optimal design treats a smaller fraction of units at the beginning and a greater share at the end.

Suggested Citation

  • Ruoxuan Xiong & Susan Athey & Mohsen Bayati & Guido Imbens, 2019. "Optimal Experimental Design for Staggered Rollouts," Papers 1911.03764, arXiv.org.
  • Handle: RePEc:arx:papers:1911.03764
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1911.03764
    File Function: Latest version
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Susan Athey & Scott Stern, 2002. "The Impact of Information Technology on Emergency Health Care Outcomes," RAND Journal of Economics, The RAND Corporation, vol. 33(3), pages 399-432, Autumn.
    2. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2017. "Matrix Completion Methods for Causal Panel Data Models," Papers 1710.10251, arXiv.org, revised Sep 2018.
    3. Jushan Bai & Serena Ng, 2002. "Determining the Number of Factors in Approximate Factor Models," Econometrica, Econometric Society, vol. 70(1), pages 191-221, January.
    4. DiMasi, Joseph A. & Grabowski, Henry G. & Hansen, Ronald W., 2016. "Innovation in the pharmaceutical industry: New estimates of R&D costs," Journal of Health Economics, Elsevier, vol. 47(C), pages 20-33.
    5. Dmitry Arkhangelsky & Susan Athey & David A. Hirshberg & Guido W. Imbens & Stefan Wager, 2019. "Synthetic Difference in Differences," Working Papers wp2019_1907, CEMFI.
    6. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    7. Varnell, S.P. & Murray, D.M. & Janega, J.B. & Blitstein, J.L., 2004. "Design and Analysis of Group-Randomized Trials: A Review of Recent Practices," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 393-399.
    8. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    9. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    10. Lawrie, Jock & Carlin, John B. & Forbes, Andrew B., 2015. "Optimal stepped wedge designs," Statistics & Probability Letters, Elsevier, vol. 99(C), pages 210-214.
    11. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    12. Susan Athey & Scott Stern, 1998. "An Empirical Framework for Testing Theories About Complimentarity in Organizational Design," NBER Working Papers 6600, National Bureau of Economic Research, Inc.
    13. Athey, Susan & Imbens, Guido W., 2018. "Design-based Analysis in Difference-In-Differences Settings with Staggered Adoption," Research Papers 3712, Stanford University, Graduate School of Business.
    14. Robert P. Leone, 1995. "Generalizing What Is Known About Temporal Aggregation and Advertising Carryover," Marketing Science, INFORMS, vol. 14(3_supplem), pages 141-150.
    15. Li, Fan & Turner, Elizabeth L. & Preisser, John S., 2018. "Optimal allocation of clusters in cohort stepped wedge designs," Statistics & Probability Letters, Elsevier, vol. 137(C), pages 257-263.
    16. Sinan Aral & Dylan Walker, 2014. "Tie Strength, Embeddedness, and Social Influence: A Large-Scale Networked Experiment," Management Science, INFORMS, vol. 60(6), pages 1352-1370, June.
    17. Leahey, T.M. & Thomas, G. & Fava, J.L. & Subak, L.L. & Schembri, M. & Krupel, K. & Kumar, R. & Weinberg, B. & Wing, R.R., 2014. "Adding evidence-based behavioral weight loss strategies to a statewide wellness campaign: A randomized clinical trial," American Journal of Public Health, American Public Health Association, vol. 104(7), pages 1300-1306.
    18. Murray, D.M. & Varnell, S.P. & Blitstein, J.L., 2004. "Design and Analysis of Group-Randomized Trials: A Review of Recent Methodological Developments," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 423-432.
    19. Li, Kathleen T. & Bell, David R., 2017. "Estimation of average treatment effects with panel data: Asymptotic theory and implementation," Journal of Econometrics, Elsevier, vol. 197(1), pages 65-75.
    20. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    21. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    22. Dimitris Bertsimas & Mac Johnson & Nathan Kallus, 2015. "The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples," Operations Research, INFORMS, vol. 63(4), pages 868-876, August.
    23. Jushan Bai, 2003. "Inferential Theory for Factor Models of Large Dimensions," Econometrica, Econometric Society, vol. 71(1), pages 135-171, January.
    24. Randall A. Lewis & Justin M. Rao, 2015. "The Unfavorable Economics of Measuring the Returns to Advertising," The Quarterly Journal of Economics, Oxford University Press, vol. 130(4), pages 1941-1973.
    25. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1911.03764. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (arXiv administrators). General contact details of provider: http://arxiv.org/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.