IDEAS home Printed from https://ideas.repec.org/p/cpr/ceprdp/11077.html
   My bibliography  Save this paper

Using Split Samples to Improve Inference on Causal Effects

Author

Listed:
  • Fafchamps, Marcel
  • Labonne, Julien

Abstract

We discuss a method aimed at reducing the risk that spurious results are published. Researchers send their datasets to an independent third party who randomly generates training and testing samples. Researchers perform their analysis on the former and once the paper is accepted for publication the method is applied to the latter and it is those results that are published. Simulations indicate that, under empirically relevant settings, the proposed method significantly reduces type I error and delivers adequate power. The method ? that can be combined with pre-analysis plans ? reduces the risk that relevant hypotheses are left untested.

Suggested Citation

  • Fafchamps, Marcel & Labonne, Julien, 2016. "Using Split Samples to Improve Inference on Causal Effects," CEPR Discussion Papers 11077, C.E.P.R. Discussion Papers.
  • Handle: RePEc:cpr:ceprdp:11077
    as

    Download full text from publisher

    File URL: https://cepr.org/publications/DP11077
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Yoav Benjamini & Abba M. Krieger & Daniel Yekutieli, 2006. "Adaptive linear step-up procedures that control the false discovery rate," Biometrika, Biometrika Trust, vol. 93(3), pages 491-507, September.
    2. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, Oxford University Press, vol. 127(4), pages 1755-1812.
    3. Eva Vivalt, 0. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    4. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    5. Angrist, Joshua D & Krueger, Alan B, 1995. "Split-Sample Instrumental Variables Estimates of the Return to Schooling," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(2), pages 225-235, April.
    6. Gerber, Alan & Malhotra, Neil, 2008. "Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals," Quarterly Journal of Political Science, now publishers, vol. 3(3), pages 313-326, October.
    7. Joshua D. Angrist & Alan B. Krueger, 1993. "Split Sample Instrumental Variables," Working Papers 699, Princeton University, Department of Economics, Industrial Relations Section..
    8. Athey, Susan & Imbens, Guido W., 2015. "Machine Learning for Estimating Heterogeneous Causal Effects," Research Papers 3350, Stanford University, Graduate School of Business.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    2. Nicola Fuchs‐Schündeln & Paolo Masella & Hannah Paule‐Paludkiewicz, 2020. "Cultural Determinants of Household Saving Behavior," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(5), pages 1035-1070, August.
    3. Clare Leaver & Owen Ozier & Pieter Serneels & Andrew Zeitlin, 2021. "Recruitment, Effort, and Retention Effects of Performance Contracts for Civil Servants: Experimental Evidence from Rwandan Primary Schools," American Economic Review, American Economic Association, vol. 111(7), pages 2213-2246, July.
    4. Haseeb, Muhammad & Vyborny, Kate, 2022. "Data, discretion and institutional capacity: Evidence from cash transfers in Pakistan," Journal of Public Economics, Elsevier, vol. 206(C).
    5. Michael L. Anderson & Jeremy Magruder, 2017. "Split-Sample Strategies for Avoiding False Discoveries," NBER Working Papers 23544, National Bureau of Economic Research, Inc.
    6. Hu, Juncheng, 2021. "Do facilitation payments affect earnings management? Evidence from China," Journal of Corporate Finance, Elsevier, vol. 68(C).
    7. Alpino, Matteo & Hauge, Karen Evelyn & Kotsadam, Andreas & Markussen, Simen, 2022. "Effects of dialogue meetings on sickness absence—Evidence from a large field experiment," Journal of Health Economics, Elsevier, vol. 83(C).
    8. Muhammad Haseeb & Kate Vyborny, 2016. "Imposing institutions: Evidence from cash transfer reform in Pakistan," CSAE Working Paper Series 2016-36, Centre for the Study of African Economies, University of Oxford.
    9. Edward Miguel, 2021. "Evidence on Research Transparency in Economics," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 193-214, Summer.
    10. Sarah A. Janzen & Jeffrey D. Michler, 2021. "Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(4), pages 1286-1304, December.
    11. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    12. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    13. Owen Ozier, 2021. "Replication Redux: The Reproducibility Crisis and the Case of Deworming [Economics of Mass Deworming Programs]," The World Bank Research Observer, World Bank Group, vol. 36(1), pages 101-130.
    14. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    15. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    16. Josephson, Anna & Michler, Jeffrey D., 2018. "Viewpoint: Beasts of the field? Ethics in agricultural and applied economics," Food Policy, Elsevier, vol. 79(C), pages 1-11.
    17. Biewen, Martin & Fitzenberger, Bernd & Seckler, Matthias, 2020. "Counterfactual quantile decompositions with selection correction taking into account Huber/Melly (2015): An application to the German gender wage gap," Labour Economics, Elsevier, vol. 67(C).
    18. Marcel Fafchamps & Julien Labonne, 2020. "Family Networks and Distributive Politics," Journal of the European Economic Association, European Economic Association, vol. 18(4), pages 1697-1725.
    19. Carol Hsu & Jae-Nam Lee & Yulin Fang & Detmar W. Straub & Ning Su & Hyun-Sun Ryu, 2022. "The Role of Vendor Legitimacy in IT Outsourcing Performance: Theory and Evidence," Information Systems Research, INFORMS, vol. 33(1), pages 337-361, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marcel Fafchamps & Julien Labonne, 2016. "Using Split Samples to Improve Inference about Causal Effects," NBER Working Papers 21842, National Bureau of Economic Research, Inc.
    2. Michael L. Anderson & Jeremy Magruder, 2017. "Split-Sample Strategies for Avoiding False Discoveries," NBER Working Papers 23544, National Bureau of Economic Research, Inc.
    3. Michele Cantarella & Chiara Strozzi, 2021. "Workers in the crowd: the labor market impact of the online platform economy [An evaluation of instrumental variable strategies for estimating the effects of catholic schooling]," Industrial and Corporate Change, Oxford University Press, vol. 30(6), pages 1429-1458.
    4. Sandberg, Susanne & Sui, Sui & Baum, Matthias, 2019. "Effects of prior market experiences and firm-specific resources on developed economy SMEs' export exit from emerging markets: Complementary or compensatory?," Journal of Business Research, Elsevier, vol. 98(C), pages 489-502.
    5. Marcello Estevao & Stacey Tevlin, 2000. "Do firms share their success with workers? The response of wages to product market conditions," Finance and Economics Discussion Series 2000-17, Board of Governors of the Federal Reserve System (U.S.).
    6. Natallia Gray & Gabriel Picone, 2018. "Evidence of Large-Scale Social Interactions in Mammography in the United States," Atlantic Economic Journal, Springer;International Atlantic Economic Society, vol. 46(4), pages 441-457, December.
    7. A. Belloni & D. Chen & V. Chernozhukov & C. Hansen, 2012. "Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain," Econometrica, Econometric Society, vol. 80(6), pages 2369-2429, November.
    8. Marco Manacorda & Andrea Tesei, 2020. "Liberation Technology: Mobile Phones and Political Mobilization in Africa," Econometrica, Econometric Society, vol. 88(2), pages 533-567, March.
    9. Lemos Sara, 2005. "Political Variables as Instruments for the Minimum Wage," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 4(1), pages 1-33, December.
    10. Abdurrahman Aydemir & George J. Borjas, 2011. "Attenuation Bias in Measuring the Wage Impact of Immigration," Journal of Labor Economics, University of Chicago Press, vol. 29(1), pages 69-113, January.
    11. Cruces, Guillermo & Galiani, Sebastian, 2007. "Fertility and female labor supply in Latin America: New causal evidence," Labour Economics, Elsevier, vol. 14(3), pages 565-573, June.
    12. Doko Tchatoka, Firmin & Dufour, Jean-Marie, 2020. "Exogeneity tests, incomplete models, weak identification and non-Gaussian distributions: Invariance and finite-sample distributional theory," Journal of Econometrics, Elsevier, vol. 218(2), pages 390-418.
    13. DUFOUR, Jean-Marie & JASIAK, Joanna, 1998. "Finite-Sample Inference Methods for Simultaneous Equations and Models with Unobserved and Generated Regressors," Cahiers de recherche 9812, Universite de Montreal, Departement de sciences economiques.
    14. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    15. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    16. Isaiah Andrews & Timothy B. Armstrong, 2017. "Unbiased instrumental variables estimation under known first‐stage sign," Quantitative Economics, Econometric Society, vol. 8(2), pages 479-503, July.
    17. Megan T Stevenson, 2018. "Distortion of Justice: How the Inability to Pay Bail Affects Case Outcomes," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 34(4), pages 511-542.
    18. Patrick Bayer & Fernando Ferreira & Stephen L. Ross, 2018. "What Drives Racial and Ethnic Differences in High-Cost Mortgages? The Role of High-Risk Lenders," Review of Financial Studies, Society for Financial Studies, vol. 31(1), pages 175-205.
    19. Kirill Borusyak & Peter Hull & Xavier Jaravel, 2022. "Quasi-Experimental Shift-Share Research Designs [Sampling-based vs. Design-based Uncertainty in Regression Analysis]," Review of Economic Studies, Oxford University Press, vol. 89(1), pages 181-213.
    20. Bernheim, B. Douglas & Garrett, Daniel M. & Maki, Dean M., 2001. "Education and saving:: The long-term effects of high school financial curriculum mandates," Journal of Public Economics, Elsevier, vol. 80(3), pages 435-465, June.

    More about this item

    Keywords

    Bonferroni correction; Data mining; Pre-analysis plan; Publication bias;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:11077. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://www.cepr.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.