IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/as9zd.html

Cherry Picking

Author

Listed:
  • Lang, Megan

    (The Abdul Latif Jameel Poverty Action Lab)

  • Qiu, Wenfeng

Abstract

Measures like pre-analysis plans ask researchers to describe planned data collection and justify data exclusions, but they provide little enforceable oversight of primary data collection. We show that a simple algorithm can select large subsets of data that yield economically meaningful and statistically significant treatment effects. The subsets cannot be distinguished from a random sample of the original data, rendering the selection undetectable if peer reviewers are unaware of the size of the original dataset. Our results hold using simulated data and replication data from a well-known study. We show that there are few natural deterrents to dataset manipulation: the results in our selected subset are robust to a range of alternative specifications, our algorithm performs well under complex sampling strategies, and our subset can yield artificially high effects on multiple outcomes. We conclude by proposing a measure to prevent such manipulation in field experiments.

Suggested Citation

  • Lang, Megan & Qiu, Wenfeng, 2021. "Cherry Picking," MetaArXiv as9zd, Center for Open Science.
    • Handle: RePEc:osf:metaar:as9zd
    DOI: 10.31219/osf.io/as9zd
    as

    Download full text from publisher

    File URL: https://osf.io/download/61256d816a7f6d001f47ab8a/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/as9zd?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Manuela Angelucci & Dean Karlan & Jonathan Zinman, 2015. "Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 151-182, January.
    2. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    3. Lenz, Gabriel S. & Sahn, Alexander, 2021. "Achieving Statistical Significance with Control Variables and Without Transparency," Political Analysis, Cambridge University Press, vol. 29(3), pages 356-369, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:osf:metaar:as9zd_v1 is not listed on IDEAS
    2. Aubry, Amandine & Héricourt, Jérôme & Marchal, Léa & Nedoncelle, Clément, 2022. "Does Immigration AffectWages? A Meta-Analysis," CEPREMAP Working Papers (Docweb) 2202, CEPREMAP.
    3. Lucia Dalla Pellegrina & Giorgio Di Maio & Paolo Landoni & Emanuele Rusinà, 2021. "Money management and entrepreneurial training in microfinance: impact on beneficiaries and institutions," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 38(3), pages 1049-1085, October.
    4. Teresa Molina Millán & Karen Macours, 2017. "Attrition in randomized control trials: Using tracking information to correct bias," FEUNL Working Paper Series novaf:wp1702, Universidade Nova de Lisboa, Faculdade de Economia.
    5. Emily Breza & Cynthia Kinnan, 2021. "Measuring the Equilibrium Impacts of Credit: Evidence from the Indian Microfinance Crisis," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 136(3), pages 1447-1497.
    6. Sergio Ocampo & Juan Herreño, 2023. "The Macroeconomic Consequences of Subsistence Self-Employment," University of Western Ontario, Departmental Research Report Series 20231, University of Western Ontario, Department of Economics.
    7. N'dri, Lasme Mathieu & Kakinaka, Makoto, 2020. "Financial inclusion, mobile money, and individual welfare: The case of Burkina Faso," Telecommunications Policy, Elsevier, vol. 44(3).
    8. Holla,Alaka & Bendini,Maria Magdalena & Dinarte Diaz,Lelys Ileana & Trako,Iva, 2021. "Is Investment in Preprimary Education Too Low ? Lessons from (Quasi) ExperimentalEvidence across Countries," Policy Research Working Paper Series 9723, The World Bank.
    9. repec:osf:osfxxx:sw6kd_v1 is not listed on IDEAS
    10. repec:osf:osfxxx:nwp8k_v1 is not listed on IDEAS
    11. Daniel Bjorkegren & Joshua Blumenstock & Omowunmi Folajimi-Senjobi & Jacqueline Mauro & Suraj R. Nair, 2022. "Instant Loans Can Lift Subjective Well-Being: A Randomized Evaluation of Digital Credit in Nigeria," Papers 2202.13540, arXiv.org.
    12. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    13. Gonzalo Haro-Álvarez & Ariadna Hern�ndez-Rivera, 2021. "Cohesión social en créditos grupales: cumplidos, regulares e incumplidos," Revista Sociedad y Economía, Universidad del Valle, CIDSE, issue 44.
    14. Bruns, Stephan & Herwartz, Helmut & Ioannidis, John P.A. & Islam, Chris-Gabriel & Raters, Fabian H. C., 2023. "Statistical reporting errors in economics," MetaArXiv mbx62, Center for Open Science.
    15. Anna Sokolova, 2023. "Marginal Propensity to Consume and Unemployment: a Meta-analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 813-846, December.
    16. Augsburg, Britta & Malde, Bansi & Olorenshaw, Harriet & Wahhaj, Zaki, 2023. "To invest or not to invest in sanitation: The role of intra-household gender differences in perceptions and bargaining power," Journal of Development Economics, Elsevier, vol. 162(C).
    17. Huber, Christoph & König-Kersting, Christian & Marini, Matteo M., 2025. "Experimenting with financial professionals," Journal of Banking & Finance, Elsevier, vol. 170(C).
    18. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    19. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    20. Gonzalez-Jimenez, David & Capozza, Francesco & Dirkmaat, Thomas & van de Veer, Evelien & van Druten, Amber & Baillon, Aurélien, 2025. "Falling and failing (to learn): Evidence from a nation-wide cybersecurity field experiment with SMEs," Journal of Economic Behavior & Organization, Elsevier, vol. 230(C).
    21. Dzemski, Andreas & Okui, Ryo & Wang, Wenjie, 2025. "Location Characteristics of Conditional Selective Confidence Intervals via Polyhedral Methods," Working Papers in Economics 851, University of Gothenburg, Department of Economics.
    22. Wenjie Wang & Yichong Zhang, 2021. "Wild Bootstrap for Instrumental Variables Regressions with Weak and Few Clusters," Papers 2108.13707, arXiv.org, revised Jan 2024.
    23. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:as9zd. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.