IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2206.04643.html
   My bibliography  Save this paper

On the Performance of the Neyman Allocation with Small Pilots

Author

Listed:
  • Yong Cai
  • Ahnaf Rafi

Abstract

The Neyman Allocation is used in many papers on experimental design, which typically assume that researchers have access to large pilot studies. This may be unrealistic. To understand the properties of the Neyman Allocation with small pilots, we study its behavior in an asymptotic framework that takes pilot size to be fixed even as the size of the main wave tends to infinity. Our analysis shows that the Neyman Allocation can lead to estimates of the ATE with higher asymptotic variance than with (non-adaptive) balanced randomization. In particular, this happens when the outcome variable is relatively homoskedastic with respect to treatment status or when it exhibits high kurtosis. We provide a series of empirical examples showing that such situations can arise in practice. Our results suggest that researchers with small pilots should not use the Neyman Allocation if they believe that outcomes are homoskedastic or heavy-tailed. Finally, we examine some potential methods for improving the finite sample performance of the FNA via simulations.

Suggested Citation

  • Yong Cai & Ahnaf Rafi, 2022. "On the Performance of the Neyman Allocation with Small Pilots," Papers 2206.04643, arXiv.org, revised Jun 2024.
    • Handle: RePEc:arx:papers:2206.04643
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2206.04643
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jinyong Hahn & Keisuke Hirano & Dean Karlan, 2011. "Adaptive Experimental Design Using the Propensity Score," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(1), pages 96-108, January.
    2. Gharad Bryan & Dean Karlan & Jonathan Zinman, 2015. "Referrals: Peer Screening and Enforcement in a Consumer Credit Field Experiment," American Economic Journal: Microeconomics, American Economic Association, vol. 7(3), pages 174-204, August.
    3. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    4. David McKenzie, 2017. "Identifying and Spurring High-Growth Entrepreneurship: Experimental Evidence from a Business Plan Competition," American Economic Review, American Economic Association, vol. 107(8), pages 2278-2307, August.
    5. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 121(2), pages 635-672.
    6. Max Cytrynbaum, 2021. "Optimal Stratification of Survey Experiments," Papers 2111.08157, arXiv.org, revised Aug 2023.
    7. Vaart,A. W. van der, 2000. "Asymptotic Statistics," Cambridge Books, Cambridge University Press, number 9780521784504.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cai, Yong & Rafi, Ahnaf, 2024. "On the performance of the Neyman Allocation with small pilots," Journal of Econometrics, Elsevier, vol. 242(1).
    2. Aufenanger, Tobias, 2017. "Machine learning to improve experimental design," FAU Discussion Papers in Economics 16/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, revised 2017.
    3. Rachel Cassidy, 2018. "Are the poor so present-biased?," IFS Working Papers W18/24, Institute for Fiscal Studies.
    4. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    5. Rachel Cassidy, 2018. "Are the poor so present-biased?," CSAE Working Paper Series 2018-19, Centre for the Study of African Economies, University of Oxford.
    6. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    7. Beaman, Lori & Karlan, Dean S. & Thuysbaert, Bram, 2014. "Saving for a (not so) Rainy Day: A Randomized Evaluation of Savings Groups in Mali," Center Discussion Papers 187189, Yale University, Economic Growth Center.
    8. Yusuke Narita, 2018. "Toward an Ethical Experiment," Cowles Foundation Discussion Papers 2127, Cowles Foundation for Research in Economics, Yale University.
    9. Guido Friebel & Matthias Heinz & Mitchell Hoffman & Nick Zubanov, 2023. "What Do Employee Referral Programs Do? Measuring the Direct and Overall Effects of a Management Practice," Journal of Political Economy, University of Chicago Press, vol. 131(3), pages 633-686.
    10. Tahir Andrabi & Jishnu Das & Asim I. Khwaja & Selcuk Ozyurt & Niharika Singh, 2020. "Upping the Ante: The Equilibrium Effects of Unconditional Grants to Private Schools," American Economic Review, American Economic Association, vol. 110(10), pages 3315-3349, October.
    11. James Andreoni & Michael Callen & Karrar Hussain & Muhammad Yasir Khan & Charles Sprenger, 2023. "Using Preference Estimates to Customize Incentives: An Application to Polio Vaccination Drives in Pakistan," Journal of the European Economic Association, European Economic Association, vol. 21(4), pages 1428-1477.
    12. Yusuke Narita, 2018. "Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials," Cowles Foundation Discussion Papers 2127r, Cowles Foundation for Research in Economics, Yale University, revised May 2019.
    13. G�nther Fink & Margaret McConnell & Sebastian Vollmer, 2014. "Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(1), pages 44-57, January.
    14. Aufenanger, Tobias, 2018. "Treatment allocation for linear models," FAU Discussion Papers in Economics 14/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, revised 2018.
    15. Meredith, Jennifer & Robinson, Jonathan & Walker, Sarah & Wydick, Bruce, 2013. "Keeping the doctor away: Experimental evidence on investment in preventative health products," Journal of Development Economics, Elsevier, vol. 105(C), pages 196-210.
    16. Liang Jiang & Xiaobin Liu & Peter C. B. Phillips & Yichong Zhang, 2024. "Bootstrap Inference for Quantile Treatment Effects in Randomized Experiments with Matched Pairs," The Review of Economics and Statistics, MIT Press, vol. 106(2), pages 542-556, March.
    17. Pascaline Dupas & Jonathan Robinson, 2013. "Savings Constraints and Microenterprise Development: Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 5(1), pages 163-192, January.
    18. Brune, Lasse & Gine, Xavier & Goldberg, Jessica & Yang, Dean, 2011. "Commitments to save : a field experiment in rural Malawi," Policy Research Working Paper Series 5748, The World Bank.
    19. Oliver Himmler & Robert Jäckle & Philipp Weinschenk, 2019. "Soft Commitments, Reminders, and Academic Performance," American Economic Journal: Applied Economics, American Economic Association, vol. 11(2), pages 114-142, April.
    20. Max Tabord-Meehan, 2023. "Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2646-2673.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2206.04643. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.