IDEAS home Printed from https://ideas.repec.org/p/ifs/cemmap/73-18.html
   My bibliography  Save this paper

Inference on winners

Author

Listed:
  • Isaiah Andrews

    (Institute for Fiscal Studies and Harvard University)

  • Toru Kitagawa

    (Institute for Fiscal Studies and University College London)

  • Adam McCloskey

    (Institute for Fiscal Studies and Brown University)

Abstract

Many empirical questions can be cast as inference on a parameter selected through optimization. For example, researchers may be interested in the effectiveness of the best policy found in a randomized trial, or the best-performing investment strategy based on historical data. Such settings give rise to a winner’s curse, where conventional estimates are biased and conventional confidence intervals are unreliable. This paper develops optimal confidence sets and median-unbiased estimators that are valid conditional on the parameter selected and so overcome this winner’s curse. If one requires validity only on average over target parameters that might have been selected, we develop hybrid procedures that combine conditional and projection confidence sets to offer further performance gains relative to existing alternatives.

Suggested Citation

  • Isaiah Andrews & Toru Kitagawa & Adam McCloskey, 2018. "Inference on winners," CeMMAP working papers CWP73/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    • Handle: RePEc:ifs:cemmap:73/18
    as

    Download full text from publisher

    File URL: https://www.ifs.org.uk/uploads/cemmap/wps/CWP731818.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    2. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    3. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1163-1228.
    4. Isaiah Andrews & Jonathan Roth & Ariel Pakes, 2023. "Inference for Linear Conditional Moment Inequalities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(6), pages 2763-2791.
    5. Keisuke Hirano & Jack R. Porter, 2009. "Asymptotics for Statistical Treatment Rules," Econometrica, Econometric Society, vol. 77(5), pages 1683-1701, September.
    6. Victor Chernozhukov & Sokbae Lee & Adam M. Rosen, 2013. "Intersection Bounds: Estimation and Inference," Econometrica, Econometric Society, vol. 81(2), pages 667-737, March.
    7. Susanne M. Schennach & Daniel Wilhelm, 2017. "A Simple Parametric Model Selection Test," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1663-1674, October.
    8. Gregory Cox, 2018. "Almost Sure Uniqueness of a Global Minimum Without Convexity," Papers 1803.02415, arXiv.org, revised Feb 2019.
    9. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    10. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility I: Childhood Exposure Effects," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1107-1162.
    11. Elliott, Graham & Muller, Ulrich K., 2007. "Confidence sets for the date of a single break in linear time series regressions," Journal of Econometrics, Elsevier, vol. 141(2), pages 1196-1218, December.
    12. Laber, Eric B. & Murphy, Susan A., 2011. "Adaptive Confidence Intervals for the Test Error in Classification," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 904-913.
    13. Sarah Baird & Joan Hamory Hicks & Michael Kremer & Edward Miguel, 2016. "Worms at Work: Long-run Impacts of a Child Health Investment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(4), pages 1637-1680.
    14. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    15. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2014. "A Practical Two‐Step Method for Testing Moment Inequalities," Econometrica, Econometric Society, vol. 82, pages 1979-2002, September.
    16. David Card & Alexandre Mas & Jesse Rothstein, 2008. "Tipping and the Dynamics of Segregation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 177-218.
    17. Abhijit Banerjee & Rema Hanna & Jordan Kyle & Benjamin A. Olken & Sudarno Sumarto, 2018. "Tangible Information and Citizen Empowerment: Identification Cards and Food Subsidy Programs in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 126(2), pages 451-491.
    18. Bhattacharya, Debopam, 2009. "Inferring Optimal Peer Assignment From Experimental Data," Journal of the American Statistical Association, American Statistical Association, vol. 104(486), pages 486-500.
    19. Dean Karlan & John A. List, 2007. "Does Price Matter in Charitable Giving? Evidence from a Large-Scale Natural Field Experiment," American Economic Review, American Economic Association, vol. 97(5), pages 1774-1793, December.
    20. Tetenov, Aleksey, 2012. "Statistical treatment choice based on asymmetric minimax regret criteria," Journal of Econometrics, Elsevier, vol. 166(1), pages 157-165.
    21. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    22. McCloskey, Adam, 2017. "Bonferroni-based size-correction for nonstandard testing problems," Journal of Econometrics, Elsevier, vol. 200(1), pages 17-35.
    23. Elliott, Graham & Müller, Ulrich K., 2014. "Pre and post break parameter inference," Journal of Econometrics, Elsevier, vol. 180(2), pages 141-157.
    24. Bruce E. Hansen, 2000. "Sample Splitting and Threshold Estimation," Econometrica, Econometric Society, vol. 68(3), pages 575-604, May.
    25. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    26. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    27. Adnan Q. Khan & Asim I. Khwaja & Benjamin A. Olken, 2016. "Tax Farming Redux: Experimental Evidence on Performance Pay for Tax Collectors," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(1), pages 219-271.
    28. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    29. Martina Björkman Nyqvist & Seema Jayachandran, 2017. "Mothers Care More, but Fathers Decide: Educating Parents about Child Health in Uganda," American Economic Review, American Economic Association, vol. 107(5), pages 496-500, May.
    30. Bryan S. Graham & Guido W. Imbens & Geert Ridder, 2014. "Complementarity and aggregate implications of assortative matching: A nonparametric analysis," Quantitative Economics, Econometric Society, vol. 5, pages 29-66, March.
    31. Andrews, Donald W.K. & Cheng, Xu & Guggenberger, Patrik, 2020. "Generic results for establishing the asymptotic size of confidence sets and tests," Journal of Econometrics, Elsevier, vol. 218(2), pages 496-531.
    32. Bachoc, Francois & Leeb, Hannes & Pötscher, Benedikt M., 2014. "Valid confidence intervals for post-model-selection predictors," MPRA Paper 60643, University Library of Munich, Germany.
    33. Esther Duflo & Michael Greenstone & Rohini Pande & Nicholas Ryan, 2018. "The Value of Regulatory Discretion: Estimates From Environmental Inspections in India," Econometrica, Econometric Society, vol. 86(6), pages 2123-2160, November.
    34. Xiaoxia Shi, 2015. "A nondegenerate Vuong test," Quantitative Economics, Econometric Society, vol. 6(1), pages 85-121, March.
    35. Bruce E. Hansen, 2001. "The New Econometrics of Structural Change: Dating Breaks in U.S. Labour Productivity," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 117-128, Fall.
    36. Howard S. Bloom & Larry L. Orr & Stephen H. Bell & George Cave & Fred Doolittle & Winston Lin & Johannes M. Bos, 1997. "The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 549-576.
    37. Yoonseok Lee & Yulong Wang, 2020. "Inference in Threshold Models," Center for Policy Research Working Papers 223, Center for Policy Research, Maxwell School, Syracuse University.
    38. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448, arXiv.org, revised May 2022.
    39. Hansen Bruce E., 1997. "Inference in TAR Models," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 2(1), pages 1-16, April.
    40. Raj Chetty & John N. Friedman & Nathaniel Hendren & Maggie R. Jones & Sonya R. Porter, 2018. "The Opportunity Atlas: Mapping the Childhood Roots of Social Mobility," NBER Working Papers 25147, National Bureau of Economic Research, Inc.
    41. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    2. Will Davis & Alexander Gordan & Rusty Tchernis, 2021. "Measuring the spatial distribution of health rankings in the United States," Health Economics, John Wiley & Sons, Ltd., vol. 30(11), pages 2921-2936, November.
    3. Dominic Coey & Kenneth Hung, 2022. "Empirical Bayes Selection for Value Maximization," Papers 2210.03905, arXiv.org, revised Jan 2023.
    4. Sergei Bazylik & Magne Mogstad & Joseph P. Romano & Azeem Shaikh & Daniel Wilhelm, 2021. "Finite- and Large-Sample Inference for Ranks using Multinomial Data with an Application to Ranking Political Parties," NBER Working Papers 29519, National Bureau of Economic Research, Inc.
    5. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    6. Kohei Yata, 2021. "Optimal Decision Rules Under Partial Identification," Papers 2111.04926, arXiv.org, revised Aug 2023.
    7. Aparajithan Venkateswaran & Anirudh Sankar & Arun G. Chandrasekhar & Tyler H. McCormick, 2024. "Robustly estimating heterogeneity in factorial data using Rashomon Partitions," Papers 2404.02141, arXiv.org, revised Aug 2024.
    8. Davide Viviano & Jelena Bradic, 2020. "Fair Policy Targeting," Papers 2005.12395, arXiv.org, revised Jun 2022.
    9. Isaiah Andrews & Jonathan Roth & Ariel Pakes, 2023. "Inference for Linear Conditional Moment Inequalities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(6), pages 2763-2791.
    10. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    11. Hsieh, Yu-Wei & Shi, Xiaoxia & Shum, Matthew, 2022. "Inference on estimators defined by mathematical programming," Journal of Econometrics, Elsevier, vol. 226(2), pages 248-268.
    12. David J. Deming, 2021. "The Growing Importance of Decision-Making on the Job," NBER Working Papers 28733, National Bureau of Economic Research, Inc.
    13. Wei, Waverly & Zhou, Yuqing & Zheng, Zeyu & Wang, Jingshen, 2024. "Inference on the best policies with many covariates," Journal of Econometrics, Elsevier, vol. 239(2).
    14. Jiafeng Chen, 2021. "Nonparametric Treatment Effect Identification in School Choice," Papers 2112.03872, arXiv.org, revised Oct 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    2. McCloskey, Adam, 2017. "Bonferroni-based size-correction for nonstandard testing problems," Journal of Econometrics, Elsevier, vol. 200(1), pages 17-35.
    3. Eric Mbakop & Max Tabord‐Meehan, 2021. "Model Selection for Treatment Choice: Penalized Welfare Maximization," Econometrica, Econometric Society, vol. 89(2), pages 825-848, March.
    4. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    5. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    6. Toru Kitagawa & Aleksey Tetenov, 2017. "Equality-minded treatment choice," CeMMAP working papers 10/17, Institute for Fiscal Studies.
    7. Toru Kitagawa & Aleksey Tetenov, 2021. "Equality-Minded Treatment Choice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 561-574, March.
    8. Davide Viviano, 2019. "Policy Targeting under Network Interference," Papers 1906.10258, arXiv.org, revised Apr 2024.
    9. Manski, Charles F., 2023. "Probabilistic prediction for binary treatment choice: With focus on personalized medicine," Journal of Econometrics, Elsevier, vol. 234(2), pages 647-663.
    10. Anders Bredahl Kock & Martin Thyrsgaard, 2017. "Optimal sequential treatment allocation," Papers 1705.09952, arXiv.org, revised Aug 2018.
    11. Garbero, Alessandra & Sakos, Grayson & Cerulli, Giovanni, 2023. "Towards data-driven project design: Providing optimal treatment rules for development projects," Socio-Economic Planning Sciences, Elsevier, vol. 89(C).
    12. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    13. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).
    14. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    15. Corradi, Valentina & Fosten, Jack & Gutknecht, Daniel, 2024. "Predictive ability tests with possibly overlapping models," Journal of Econometrics, Elsevier, vol. 241(1).
    16. Undral Byambadalai, 2022. "Identification and Inference for Welfare Gains without Unconfoundedness," Papers 2207.04314, arXiv.org.
    17. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    18. Toru Kitagawa & Hugo Lopez & Jeff Rowley, 2022. "Stochastic Treatment Choice with Empirical Welfare Updating," Papers 2211.01537, arXiv.org, revised Feb 2023.
    19. Debopam Bhattacharya & Pascaline Dupas & Shin Kanaya, 2013. "Estimating the Impact of Means-tested Subsidies under Treatment Externalities with Application to Anti-Malarial Bednets," CREATES Research Papers 2013-06, Department of Economics and Business Economics, Aarhus University.
    20. Davide Viviano & Kaspar Wuthrich & Paul Niehaus, 2021. "A model of multiple hypothesis testing," Papers 2104.13367, arXiv.org, revised Apr 2024.

    More about this item

    Keywords

    winner's curse; selective inference;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ifs:cemmap:73/18. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Emma Hyman (email available below). General contact details of provider: https://edirc.repec.org/data/cmifsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.