IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v50y2025i5p731-764.html

Improving the Estimation of Site-Specific Effects and Their Distribution in Multisite Trials

Author

Listed:
  • JoonHo Lee

    (The University of Alabama)

  • Jonathan Che

    (Exponent)

  • Sophia Rabe-Hesketh
  • Avi Feller

    (University of California, Berkeley)

  • Luke Miratrix

    (Harvard University)

Abstract

In multisite trials, researchers are often interested in several inferential goals: estimating treatment effects for each site, ranking these effects, and studying their distribution. This study seeks to identify optimal methods for estimating these targets. Through a comprehensive simulation study, we assess two strategies and their combined effects: semiparametric modeling of the prior distribution and alternative posterior summary methods tailored to minimize specific loss functions. Our findings highlight that the success of different estimation strategies depends largely on the amount of within-site and between-site information available from the data. We discuss how our results can guide balancing the trade-offs associated with shrinkage in limited data environments.

Suggested Citation

  • JoonHo Lee & Jonathan Che & Sophia Rabe-Hesketh & Avi Feller & Luke Miratrix, 2025. "Improving the Estimation of Site-Specific Effects and Their Distribution in Multisite Trials," Journal of Educational and Behavioral Statistics, , vol. 50(5), pages 731-764, October.
    • Handle: RePEc:sae:jedbes:v:50:y:2025:i:5:p:731-764
    DOI: 10.3102/10769986241254286
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986241254286
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986241254286?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bradley Efron, 2016. "Empirical Bayes deconvolution estimates," Biometrika, Biometrika Trust, vol. 103(1), pages 1-20.
    2. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    3. Joshua Angrist & Peter Hull & Christopher R. Walters, 2022. "Methods for Measuring School Effectiveness," NBER Working Papers 30803, National Bureau of Economic Research, Inc.
    4. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    5. Jiaying Gu & Roger Koenker, 2023. "Reply to: Comments on “Invidious Comparisons: Ranking and Selection as Compound Decisions”," Econometrica, Econometric Society, vol. 91(1), pages 61-66, January.
    6. Felipe Barrera-Osorio & Leigh L. Linden & Juan E. Saavedra, 2019. "Medium- and Long-Term Educational Consequences of Alternative Conditional Cash Transfer Designs: Experimental Evidence from Colombia," American Economic Journal: Applied Economics, American Economic Association, vol. 11(3), pages 54-91, July.
    7. Jiaying Gu & Roger Koenker, 2023. "Invidious Comparisons: Ranking and Selection as Compound Decisions," Econometrica, Econometric Society, vol. 91(1), pages 1-41, January.
    8. Sally Paganin & Christopher J. Paciorek & Claudia Wehrhahn & Abel Rodríguez & Sophia Rabe-Hesketh & Perry de Valpine, 2023. "Computational Strategies and Estimation Performance With Bayesian Semiparametric Item Response Theory Models," Journal of Educational and Behavioral Statistics, , vol. 48(2), pages 147-188, April.
    9. Rachael Meager, 2019. "Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 11(1), pages 57-91, January.
    10. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    11. Meager, Rachael, 2019. "Understanding the average impact of microcredit expansions: a Bayesian hierarchical analysis of seven randomized experiments," LSE Research Online Documents on Economics 88190, London School of Economics and Political Science, LSE Library.
    12. David I. Ohlssen & Linda D. Sharples & David J. Spiegelhalter, 2007. "A hierarchical modelling framework for identifying unusual performance in health care providers," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 865-890, October.
    13. Timothy B. Armstrong & Michal Kolesár & Mikkel Plagborg‐Møller, 2022. "Robust Empirical Bayes Confidence Intervals," Econometrica, Econometric Society, vol. 90(6), pages 2567-2602, November.
    14. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Erratum to “Leveraging Lotteries for School Value-Added: Testing and Estimation”," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(4), pages 2061-2062.
    15. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    16. Patrick Kline & Evan K Rose & Christopher R Walters, 2022. "Systemic Discrimination Among Large U.S. Employers [“Teachers and Student Achievement in the Chicago Public High Schools,”]," The Quarterly Journal of Economics, Oxford University Press, vol. 137(4), pages 1963-2036.
    17. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    18. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448, arXiv.org, revised May 2022.
    19. Wei Shen & Thomas A. Louis, 1998. "Triple‐goal estimates in two‐stage hierarchical models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 60(2), pages 455-471.
    20. Wendimagegn Ghidey & Emmanuel Lesaffre & Paul Eilers, 2004. "Smooth Random Effects Distribution in a Linear Mixed Model," Biometrics, The International Biometric Society, vol. 60(4), pages 945-953, December.
    21. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Walters, Christopher, 2024. "Empirical Bayes methods in labor economics," Handbook of Labor Economics,, Elsevier.
    2. JoonHo Lee & Daihe Sui, 2025. "Fully Bayesian Inference for Meta-Analytic Deconvolution Using Efron’s Log-Spline Prior," Mathematics, MDPI, vol. 13(16), pages 1-49, August.
    3. Bonhomme, Stéphane & Denis, Angela, 2024. "Estimating heterogeneous effects: Applications to labor economics," Labour Economics, Elsevier, vol. 91(C).
    4. Takahiro Hoshino & Makoto Nakakita, 2026. "Shared-Donor Inference for Heterogeneity in Many-Group Synthetic Difference-in-Differences," Papers 2607.08324, arXiv.org, revised Aug 2026.
    5. Raffaella Giacomini & Sokbae Lee & Silvia Sarpietro, 2023. "Individual Shrinkage for Random Effects," Papers 2308.01596, arXiv.org, revised May 2026.
    6. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    7. Juan C. Yamin, 2025. "Poverty Targeting with Imperfect Information," Papers 2506.18188, arXiv.org, revised May 2026.
    8. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    9. Sarah Moon, 2025. "Optimal Policy Choices Under Uncertainty," Papers 2503.03910, arXiv.org, revised Aug 2026.
    10. Zhiheng You, 2026. "Using Prior Studies to Design Experiments: An Empirical Bayes Approach," Papers 2602.20581, arXiv.org.
    11. Seungjin Han & Julius Owusu & Youngki Shin, 2026. "Ranking Treatment Saturations under Clustered Network Interference," Papers 2606.18590, arXiv.org.
    12. Susan Athey & Kristen Grabarz & Michael Luca & Nils Wernerfelt, 2023. "Digital public health interventions at scale: The impact of social media advertising on beliefs and outcomes related to COVID vaccines," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(5), pages 2208110120-, January.
    13. Pietro Emilio Spini, 2021. "A Simple Measure of Robustness for External Validity under Covariate Shifts," Papers 2112.09259, arXiv.org, revised May 2026.
    14. Yechan Park & Yuya Sasaki, 2026. "Choosing A Headline Estimand from Matching, DID, and Hybrid Designs: A Minimax-Regret Approach," Papers 2606.20435, arXiv.org.
    15. Raffaella Giacomini & Sokbae Lee & Silvia Sarpietro, 2023. "A Robust Method for Microforecasting and Estimation of Random Effects," Working Paper Series WP 2023-26, Federal Reserve Bank of Chicago.
    16. Denis Fougère & Nicolas Jacquemet, 2019. "Causal Inference and Impact Evaluation," Economie et Statistique / Economics and Statistics, Institut National de la Statistique et des Etudes Economiques (INSEE), issue 510-511-5, pages 181-200.
    17. Easterly, William & Pennings, Steven, 2025. "Leader value added: Assessing the growth contribution of individual national leaders," Journal of Development Economics, Elsevier, vol. 175(C).
    18. Jiaying Gu & Roger Koenker, 2020. "Invidious Comparisons: Ranking and Selection as Compound Decisions," Papers 2012.12550, arXiv.org, revised Sep 2021.
    19. Eric Auerbach & Jonathan Auerbach & Sidonia McKenzie, 2026. "Post-Selection Inference for Network Structure," Papers 2607.00312, arXiv.org, revised Jul 2026.
    20. Leonardo Iacovone & David McKenzie & Rachael Meager, 2025. "Bayesian Impact Evaluation With Informative Priors: An Application to a Colombian Management and Export Improvement Program," Econometrica, Econometric Society, vol. 93(5), pages 1915-1935, September.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:50:y:2025:i:5:p:731-764. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.