IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v50y2025i3p420-448.html

Approaches to Statistical Efficiency When Comparing the Embedded Adaptive Interventions in a SMART

Author

Listed:
  • Timothy Lycurgus

    (University of Michigan)

  • Amy Kilbourne

    (University of Michigan Medical School)

  • Daniel Almirall

    (University of Michigan)

Abstract

Sequential, multiple assignment randomized trials (SMARTs), which assist in the optimization of adaptive interventions, are growing in popularity in education and behavioral sciences. This is unsurprising, as adaptive interventions reflect the sequential, tailored nature of learning in a classroom or school. Nonetheless, as is true elsewhere in education research, observed effect sizes in education-based SMARTs are frequently small. As a consequence, statistical efficiency is of paramount importance in their analysis. The contributions of this manuscript are twofold. First, we provide an overview of adaptive interventions and SMART designs for researchers in education science. Second, we propose four techniques that have the potential to improve statistical efficiency in the analysis of SMARTs. We demonstrate the benefits of these techniques in SMART settings both through the analysis of a SMART designed to optimize an adaptive intervention for increasing cognitive behavioral therapy delivery in school settings and through a comprehensive simulation study. Each of the proposed techniques is easily implementable, either with over-the-counter statistical software or through R code provided in Supplemental Material.

Suggested Citation

  • Timothy Lycurgus & Amy Kilbourne & Daniel Almirall, 2025. "Approaches to Statistical Efficiency When Comparing the Embedded Adaptive Interventions in a SMART," Journal of Educational and Behavioral Statistics, , vol. 50(3), pages 420-448, June.
    • Handle: RePEc:sae:jedbes:v:50:y:2025:i:3:p:420-448
    DOI: 10.3102/10769986241251419
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986241251419
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986241251419?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Brumback, Babette A., 2009. "A note on using the estimated versus the known propensity score to estimate the average treatment effect," Statistics & Probability Letters, Elsevier, vol. 79(4), pages 537-542, February.
    2. Murphy, S. A. & Bingham, D., 2009. "Screening Experiments for Developing Dynamic Treatment Regimes," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 391-408.
    3. Stijn Vansteelandt, 2007. "On Confounding, Prediction and Efficiency in the Analysis of Longitudinal and Cross‐sectional Clustered Data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 34(3), pages 478-498, September.
    4. Orellana Liliana & Rotnitzky Andrea & Robins James M., 2010. "Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part II: Proofs of Results," The International Journal of Biostatistics, De Gruyter, vol. 6(2), pages 1-19, March.
    5. Orellana Liliana & Rotnitzky Andrea & Robins James M., 2010. "Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content," The International Journal of Biostatistics, De Gruyter, vol. 6(2), pages 1-49, March.
    6. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Q. Clairon & R. Henderson & N. J. Young & E. D. Wilson & C. J. Taylor, 2021. "Adaptive treatment and robust control," Biometrics, The International Biometric Society, vol. 77(1), pages 223-236, March.
    2. Jelena Bradic & Weijie Ji & Yuqian Zhang, 2021. "High-dimensional Inference for Dynamic Treatment Effects," Papers 2110.04924, arXiv.org, revised May 2023.
    3. Ying Huang & Youyi Fong, 2014. "Identifying optimal biomarker combinations for treatment selection via a robust kernel method," Biometrics, The International Biometric Society, vol. 70(4), pages 891-901, December.
    4. Ruoqing Zhu & Ying-Qi Zhao & Guanhua Chen & Shuangge Ma & Hongyu Zhao, 2017. "Greedy outcome weighted tree learning of optimal personalized treatment rules," Biometrics, The International Biometric Society, vol. 73(2), pages 391-400, June.
    5. Mélanie Prague & Daniel Commenges & Jon Michael Gran & Bruno Ledergerber & Jim Young & Hansjakob Furrer & Rodolphe Thiébaut, 2017. "Dynamic models for estimating the effect of HAART on CD4 in observational studies: Application to the Aquitaine Cohort and the Swiss HIV Cohort Study," Biometrics, The International Biometric Society, vol. 73(1), pages 294-304, March.
    6. Eric B. Laber & Anastasios A. Tsiatis & Marie Davidian & Shannon T. Holloway, 2014. "Discussion of “Combining biomarkers to optimize patient treatment recommendation”," Biometrics, The International Biometric Society, vol. 70(3), pages 707-710, September.
    7. Shu Yang & Anastasios A. Tsiatis & Michael Blazing, 2018. "Modeling survival distribution as a function of time to treatment discontinuation: A dynamic treatment regime approach," Biometrics, The International Biometric Society, vol. 74(3), pages 900-909, September.
    8. Victor Chernozhukov & Whitney Newey & Rahul Singh & Vasilis Syrgkanis, 2022. "Automatic Debiased Machine Learning for Dynamic Treatment Effects and General Nested Functionals," Papers 2203.13887, arXiv.org, revised Jun 2023.
    9. Xiaofei Bai & Anastasios A. Tsiatis & Wenbin Lu & Rui Song, 2017. "Optimal treatment regimes for survival endpoints using a locally-efficient doubly-robust estimator from a classification perspective," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 23(4), pages 585-604, October.
    10. Yasuhiro Hagiwara & Tomohiro Shinozaki & Hirofumi Mukai & Yutaka Matsuyama, 2021. "Sensitivity analysis for subsequent treatments in confirmatory oncology clinical trials: A two‐stage stochastic dynamic treatment regime approach," Biometrics, The International Biometric Society, vol. 77(2), pages 702-714, June.
    11. Yuqian Zhang & Weijie Ji & Jelena Bradic, 2021. "Dynamic treatment effects: high-dimensional inference under model misspecification," Papers 2111.06818, arXiv.org, revised Jan 2025.
    12. Yunan Wu & Lan Wang, 2021. "Resampling‐based confidence intervals for model‐free robust inference on optimal treatment regimes," Biometrics, The International Biometric Society, vol. 77(2), pages 465-476, June.
    13. Zhishuai Liu & Jesse Clifton & Eric B. Laber & John Drake & Ethan X. Fang, 2023. "Deep Spatial Q-Learning for Infectious Disease Control," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 28(4), pages 749-773, December.
    14. Eric B. Laber & Daniel J. Lizotte & Bradley Ferguson, 2014. "Set-valued dynamic treatment regimes for competing outcomes," Biometrics, The International Biometric Society, vol. 70(1), pages 53-61, March.
    15. Emily L. Butler & Eric B. Laber & Sonia M. Davis & Michael R. Kosorok, 2018. "Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules," Biometrics, The International Biometric Society, vol. 74(1), pages 18-26, March.
    16. Isaac Meza & Rahul Singh, 2021. "Nested Nonparametric Instrumental Variable Regression," Papers 2112.14249, arXiv.org, revised May 2025.
    17. Zhen Li & Jie Chen & Eric Laber & Fang Liu & Richard Baumgartner, 2023. "Optimal Treatment Regimes: A Review and Empirical Comparison," International Statistical Review, International Statistical Institute, vol. 91(3), pages 427-463, December.
    18. Cain Lauren E. & Robins James M. & Lanoy Emilie & Logan Roger & Costagliola Dominique & Hernán Miguel A., 2010. "When to Start Treatment? A Systematic Approach to the Comparison of Dynamic Regimes Using Observational Data," The International Journal of Biostatistics, De Gruyter, vol. 6(2), pages 1-26, April.
    19. Lucia Babino & Andrea Rotnitzky & James Robins, 2019. "Multiple robust estimation of marginal structural mean models for unconstrained outcomes," Biometrics, The International Biometric Society, vol. 75(1), pages 90-99, March.
    20. Baqun Zhang & Anastasios A. Tsiatis & Eric B. Laber & Marie Davidian, 2012. "A Robust Method for Estimating Optimal Treatment Regimes," Biometrics, The International Biometric Society, vol. 68(4), pages 1010-1018, December.

    More about this item

    Keywords

    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:50:y:2025:i:3:p:420-448. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.