IDEAS home Printed from https://ideas.repec.org/a/wly/emetrp/v91y2023i1p1-41.html

Invidious Comparisons: Ranking and Selection as Compound Decisions

Author

Listed:
  • Jiaying Gu
  • Roger Koenker

Abstract

There is an innate human tendency, one might call it the “league table mentality,” to construct rankings. Schools, hospitals, sports teams, movies, and myriad other objects are ranked even though their inherent multi‐dimensionality would suggest that—at best—only partial orderings were possible. We consider a large class of elementary ranking problems in which we observe noisy, scalar measurements of merit for n objects of potentially heterogeneous precision and are asked to select a group of the objects that are “most meritorious.” The problem is naturally formulated in the compound decision framework of Robbins's (1956) empirical Bayes theory, but it also exhibits close connections to the recent literature on multiple testing. The nonparametric maximum likelihood estimator for mixture models (Kiefer and Wolfowitz (1956)) is employed to construct optimal ranking and selection rules. Performance of the rules is evaluated in simulations and an application to ranking U.S. kidney dialysis centers.

Suggested Citation

  • Jiaying Gu & Roger Koenker, 2023. "Invidious Comparisons: Ranking and Selection as Compound Decisions," Econometrica, Econometric Society, vol. 91(1), pages 1-41, January.
    • Handle: RePEc:wly:emetrp:v:91:y:2023:i:1:p:1-41
    DOI: 10.3982/ECTA19304
    as

    Download full text from publisher

    File URL: https://doi.org/10.3982/ECTA19304
    Download Restriction: no
    File URL: https://libkey.io/10.3982/ECTA19304?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Heckman, James & Singer, Burton, 1984. "A Method for Minimizing the Impact of Distributional Assumptions in Econometric Models for Duration Data," Econometrica, Econometric Society, vol. 52(2), pages 271-320, March.
    2. Wenguang Sun & Alexander C. McLain, 2012. "Multiple Testing of Composite Null Hypotheses in Heteroscedastic Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 673-687, June.
    3. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1163-1228.
    4. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    5. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility I: Childhood Exposure Effects," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1107-1162.
    6. Jiaying Gu & Roger Koenker, 2017. "Empirical Bayesball Remixed: Empirical Bayes Methods for Longitudinal Data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 575-599, April.
    7. Athey, Susan & Imbens, Guido W. & Metzger, Jonas & Munro, Evan, 2024. "Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations," Journal of Econometrics, Elsevier, vol. 240(2).
    8. Sun, Wenguang & Cai, T. Tony, 2007. "Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 901-912, September.
    9. Jiaying Gu & Roger Koenker, 2017. "Rebayes: an R package for empirical bayes mixture methods," CeMMAP working papers CWP37/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    10. Michael Gilraine & Jiaying Gu & Robert McMillan, 2020. "A New Method for Estimating Teacher Value-Added," NBER Working Papers 27094, National Bureau of Economic Research, Inc.
    11. Jiaying Gu & Roger Koenker, 2017. "Rebayes: an R package for empirical bayes mixture methods," CeMMAP working papers 37/17, Institute for Fiscal Studies.
    12. Roger Koenker & Ivan Mizera, 2014. "Convex Optimization, Shape Constraints, Compound Decisions, and Empirical Bayes Rules," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(506), pages 674-685, June.
    13. Nicholas C. Henderson & Michael A. Newton, 2016. "Making the cut: improved ranking and selection for large-scale inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(4), pages 781-804, September.
    14. Hongyuan Cao & Wenguang Sun & Michael R. Kosorok, 2013. "The optimal power puzzle: scrutiny of the monotone likelihood ratio assumption in multiple testing," Biometrika, Biometrika Trust, vol. 100(2), pages 495-502.
    15. Portnoy, Stephen, 1982. "Maximizing the probability of correctly ordering random variables using linear predictors," Journal of Multivariate Analysis, Elsevier, vol. 12(2), pages 256-269, June.
    16. Koenker, Roger & Mizera, Ivan, 2014. "Convex Optimization in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 60(i05).
    17. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    18. Hanushek, Eric A., 2011. "The economic value of higher teacher quality," Economics of Education Review, Elsevier, vol. 30(3), pages 466-479, June.
    19. Christopher Genovese & Larry Wasserman, 2002. "Operating characteristics and extensions of the false discovery rate procedure," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(3), pages 499-517, August.
    20. John D. Storey, 2002. "A direct approach to false discovery rates," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(3), pages 479-498, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bonhomme, Stéphane & Denis, Angela, 2024. "Estimating heterogeneous effects: Applications to labor economics," Labour Economics, Elsevier, vol. 91(C).
    2. JoonHo Lee & Daihe Sui, 2025. "Fully Bayesian Inference for Meta-Analytic Deconvolution Using Efron’s Log-Spline Prior," Mathematics, MDPI, vol. 13(16), pages 1-49, August.
    3. Sarah Moon, 2025. "Optimal Policy Choices Under Uncertainty," Papers 2503.03910, arXiv.org, revised Feb 2026.
    4. JoonHo Lee & Jonathan Che & Sophia Rabe-Hesketh & Avi Feller & Luke Miratrix, 2025. "Improving the Estimation of Site-Specific Effects and Their Distribution in Multisite Trials," Journal of Educational and Behavioral Statistics, , vol. 50(5), pages 731-764, October.
    5. Walters, Christopher, 2024. "Empirical Bayes methods in labor economics," Handbook of Labor Economics,, Elsevier.
    6. Juan C. Yamin, 2025. "Poverty Targeting with Imperfect Information," Papers 2506.18188, arXiv.org, revised May 2026.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jiaying Gu & Roger Koenker, 2020. "Invidious Comparisons: Ranking and Selection as Compound Decisions," Papers 2012.12550, arXiv.org, revised Sep 2021.
    2. Walters, Christopher, 2024. "Empirical Bayes methods in labor economics," Handbook of Labor Economics,, Elsevier.
    3. Jiaying Gu & Roger Koenker, 2016. "On a Problem of Robbins," International Statistical Review, International Statistical Institute, vol. 84(2), pages 224-244, August.
    4. Timothy B. Armstrong & Michal Kolesár & Mikkel Plagborg‐Møller, 2022. "Robust Empirical Bayes Confidence Intervals," Econometrica, Econometric Society, vol. 90(6), pages 2567-2602, November.
    5. Jiaying Gu & Roger Koenker, 2017. "Rebayes: an R package for empirical bayes mixture methods," CeMMAP working papers CWP37/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    6. Juan C. Yamin, 2025. "Poverty Targeting with Imperfect Information," Papers 2506.18188, arXiv.org, revised May 2026.
    7. Jiaying Gu & Roger Koenker, 2017. "Rebayes: an R package for empirical bayes mixture methods," CeMMAP working papers 37/17, Institute for Fiscal Studies.
    8. Christopher R. Walters, 2024. "Empirical Bayes Methods in Labor Economics," NBER Working Papers 33091, National Bureau of Economic Research, Inc.
    9. Cipolli III, William & Hanson, Timothy & McLain, Alexander C., 2016. "Bayesian nonparametric multiple testing," Computational Statistics & Data Analysis, Elsevier, vol. 101(C), pages 64-79.
    10. T. Tony Cai & Wenguang Sun & Weinan Wang, 2019. "Covariate‐assisted ranking and screening for large‐scale two‐sample inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(2), pages 187-234, April.
    11. Alberto Abadie & Maximilian Kasy, 2019. "Choosing Among Regularized Estimators in Empirical Economics: The Risk of Machine Learning," The Review of Economics and Statistics, MIT Press, vol. 101(5), pages 743-762, December.
    12. Yeil Kwon & Zhigen Zhao, 2023. "On F-modelling-based empirical Bayes estimation of variances," Biometrika, Biometrika Trust, vol. 110(1), pages 69-81.
    13. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448, arXiv.org, revised May 2022.
    14. Gómez-Villegas Miguel A. & Salazar Isabel & Sanz Luis, 2014. "A Bayesian decision procedure for testing multiple hypotheses in DNA microarray experiments," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 13(1), pages 49-65, February.
    15. Zhiheng You, 2026. "Using Prior Studies to Design Experiments: An Empirical Bayes Approach," Papers 2602.20581, arXiv.org.
    16. Tian Xie, 2025. "Automatic Inference for Value-Added Regressions," Papers 2503.19178, arXiv.org, revised Dec 2025.
    17. Andros Kourtellos & Chih Ming Tan & Steven N. Durlauf, 2022. "The Great Gatsby Curve," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 571-605, August.
    18. Soonwoo Kwon, 2023. "Optimal Shrinkage Estimation of Fixed Effects in Linear Panel Data Models," Papers 2308.12485, arXiv.org, revised Sep 2025.
    19. Jiaying Gu & Roger Koenker, 2018. "Nonparametric maximum likelihood methods for binary response models with random coefficients," Papers 1811.03329, arXiv.org, revised Jan 2020.
    20. Jiaying Gu & Roger Koenker, 2014. "Unobserved heterogeneity in income dynamics: an empirical Bayes perspective," CeMMAP working papers 43/14, Institute for Fiscal Studies.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:emetrp:v:91:y:2023:i:1:p:1-41. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/essssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.