IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2311.14204.html
   My bibliography  Save this paper

Reproducible Aggregation of Sample-Split Statistics

Author

Listed:
  • David M. Ritzwoller
  • Joseph P. Romano

Abstract

Statistical inference is often simplified by sample-splitting. This simplification comes at the cost of the introduction of randomness not native to the data. We propose a simple procedure for sequentially aggregating statistics constructed with multiple splits of the same sample. The user specifies a bound and a nominal error rate. If the procedure is implemented twice on the same data, the nominal error rate approximates the chance that the results differ by more than the bound. We illustrate the application of the procedure to several widely applied econometric methods.

Suggested Citation

  • David M. Ritzwoller & Joseph P. Romano, 2023. "Reproducible Aggregation of Sample-Split Statistics," Papers 2311.14204, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2311.14204
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2311.14204
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Robin Dunn & Aaditya Ramdas & Sivaraman Balakrishnan & Larry Wasserman, 2023. "Gaussian universal likelihood ratio testing," Biometrika, Biometrika Trust, vol. 110(2), pages 319-337.
    2. Jiafeng Chen & David M. Ritzwoller, 2021. "Semiparametric Estimation of Long-Term Treatment Effects," Papers 2107.14405, arXiv.org, revised Aug 2023.
    3. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    4. Jing Lei, 2020. "Cross-Validation With Confidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(532), pages 1978-1997, December.
    5. Nitis Mukhopadhyay & Sujay Datta, 1996. "On sequential fixed-width confidence intervals for the mean and second-order expansions of the associated coverage probabilities," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 48(3), pages 497-507, September.
    6. Chen, Jiafeng & Ritzwoller, David M., 2023. "Semiparametric estimation of long-term treatment effects," Journal of Econometrics, Elsevier, vol. 237(2).
    7. Meinshausen, Nicolai & Meier, Lukas & Bühlmann, Peter, 2009. "p-Values for High-Dimensional Regression," Journal of the American Statistical Association, American Statistical Association, vol. 104(488), pages 1671-1681.
    8. Victor Chernozhukov & Mert Demirer & Esther Duflo & Iván Fernández-Val, 2018. "Generic Machine Learning Inference on Heterogeneous Treatment Effects in Randomized Experiments, with an Application to Immunization in India," NBER Working Papers 24678, National Bureau of Economic Research, Inc.
    9. DiCiccio, Cyrus J. & DiCiccio, Thomas J. & Romano, Joseph P., 2020. "Exact tests via multiple data splitting," Statistics & Probability Letters, Elsevier, vol. 166(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David M. Ritzwoller & Vasilis Syrgkanis, 2024. "Simultaneous Inference for Local Structural Parameters with Random Forests," Papers 2405.07860, arXiv.org, revised Sep 2024.
    2. Simon Calmar Andersen & Louise Beuchert & Phillip Heiler & Helena Skyt Nielsen, 2023. "A Guide to Impact Evaluation under Sample Selection and Missing Data: Teacher's Aides and Adolescent Mental Health," Papers 2308.04963, arXiv.org.
    3. Solari, Aldo & Djordjilović, Vera, 2022. "Multi split conformal prediction," Statistics & Probability Letters, Elsevier, vol. 184(C).
    4. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-Dimensional Econometrics and Regularized GMM," Papers 1806.01888, arXiv.org, revised Jun 2018.
    5. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    6. Xu, Yang & Zhao, Shishun & Hu, Tao & Sun, Jianguo, 2021. "Variable selection for generalized odds rate mixture cure models with interval-censored failure time data," Computational Statistics & Data Analysis, Elsevier, vol. 156(C).
    7. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    8. Sung Jae Jun & Sokbae Lee, 2024. "Causal Inference Under Outcome-Based Sampling with Monotonicity Assumptions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(3), pages 998-1009, July.
    9. Abhijit Banerjee & Emily Breza & Esther Duflo & Cynthia Kinnan, 2019. "Can Microfinance Unlock a Poverty Trap for Some Entrepreneurs?," NBER Working Papers 26346, National Bureau of Economic Research, Inc.
    10. Shi, Chengchun & Zhou, Yunzhe & Li, Lexin, 2024. "Testing directed acyclic graph via structural, supervised and generative adversarial learning," LSE Research Online Documents on Economics 119446, London School of Economics and Political Science, LSE Library.
    11. Stefano Piasenti & Marica Valente & Roel van Veldhuizen & Gregor Pfeifer & Gregor-Gabriel Pfeifer, 2023. "Does Unfairness Hurt Women? The Effects of Losing Unfair Competitions," CESifo Working Paper Series 10572, CESifo.
    12. Elek, Péter & Bíró, Anikó, 2021. "Regional differences in diabetes across Europe – regression and causal forest analyses," Economics & Human Biology, Elsevier, vol. 40(C).
    13. Michael C. Knaus, 2021. "A double machine learning approach to estimate the effects of musical practice on student’s skills," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(1), pages 282-300, January.
    14. Frölich, Markus & Michaelowa, Katharina, 2004. "Peer effects and textbooks in primary education: Evidence from francophone sub-Saharan Africa," HWWA Discussion Papers 311, Hamburg Institute of International Economics (HWWA).
    15. Wei Huang & Oliver Linton & Zheng Zhang, 2022. "A Unified Framework for Specification Tests of Continuous Treatment Effect Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1817-1830, October.
    16. Kyle Colangelo & Ying-Ying Lee, 2019. "Double debiased machine learning nonparametric inference with continuous treatments," CeMMAP working papers CWP54/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    17. Chunrong Ai & Oliver Linton & Kaiji Motegi & Zheng Zhang, 2021. "A unified framework for efficient estimation of general treatment models," Quantitative Economics, Econometric Society, vol. 12(3), pages 779-816, July.
    18. Jinyong Hahn & Keisuke Hirano & Dean Karlan, 2011. "Adaptive Experimental Design Using the Propensity Score," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(1), pages 96-108, January.
    19. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    20. Victor Chernozhukov & Ivan Fernandez-Val & Christian Hansen, 2013. "Program evaluation with high-dimensional data," CeMMAP working papers CWP57/13, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2311.14204. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.