IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0228627.html
   My bibliography  Save this article

The aggregation paradox for statistical rankings and nonparametric tests

Author

Listed:
  • Haikady N Nagaraja
  • Shane Sanders

Abstract

The relationship between social choice aggregation rules and non-parametric statistical tests has been established for several cases. An outstanding, general question at this intersection is whether there exists a non-parametric test that is consistent upon aggregation of data sets (not subject to Yule-Simpson Aggregation Paradox reversals for any ordinal data). Inconsistency has been shown for several non-parametric tests, where the property bears fundamentally upon robustness (ambiguity) of non-parametric test (social choice) results. Using the binomial(n, p = 0.5) random variable CDF, we prove that aggregation of r(≥2) constituent data sets—each rendering a qualitatively-equivalent sign test for matched pairs result—reinforces and strengthens constituent results (sign test consistency). Further, we prove that magnitude of sign test consistency strengthens in significance-level of constituent results (strong-form consistency). We then find preliminary evidence that sign test consistency is preserved for a generalized form of aggregation. Application data illustrate (in)consistency in non-parametric settings, and links with information aggregation mechanisms (as well as paradoxes thereof) are discussed.

Suggested Citation

  • Haikady N Nagaraja & Shane Sanders, 2020. "The aggregation paradox for statistical rankings and nonparametric tests," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-21, March.
    • Handle: RePEc:plo:pone00:0228627
    DOI: 10.1371/journal.pone.0228627
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0228627
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0228627&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0228627?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Alessandro Selvitella, 2017. "The ubiquity of the Simpson’s Paradox," Journal of Statistical Distributions and Applications, Springer, vol. 4(1), pages 1-16, December.
    2. Quinn McNemar, 1947. "Note on the sampling error of the difference between correlated proportions or percentages," Psychometrika, Springer;The Psychometric Society, vol. 12(2), pages 153-157, June.
    3. Datta, Somnath & Satten, Glen A., 2005. "Rank-Sum Tests for Clustered Data," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 908-915, September.
    4. Bennouri, Moez & Gimpel, Henner & Robert, Jacques, 2011. "Measuring the impact of information aggregation mechanisms: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 78(3), pages 302-318, May.
    5. Daniel McFadden & Kenneth Train, 2000. "Mixed MNL models for discrete response," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(5), pages 447-470.
    6. Vartia, Yrjo O, 1983. "Efficient Methods of Measuring Welfare Change and Compensated Income in Terms of Ordinary Demand Functions," Econometrica, Econometric Society, vol. 51(1), pages 79-98, January.
    7. Li Hao & Daniel Houser, 2015. "Adaptive Procedures for the Wilcoxon–Mann–Whitney Test: Seven Decades of Advances," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 44(9), pages 1939-1957, May.
    8. Matzkin, Rosa L, 1992. "Nonparametric and Distribution-Free Estimation of the Binary Threshold Crossing and the Binary Choice Models," Econometrica, Econometric Society, vol. 60(2), pages 239-270, March.
    9. Gur Yaari & Shmuel Eisenmann, 2011. "The Hot (Invisible?) Hand: Can Time Sequence Patterns of Success/Failure in Sports Be Modeled as Repeated Random Independent Trials?," PLOS ONE, Public Library of Science, vol. 6(10), pages 1-10, October.
    10. Saari, Donald G., 1999. "Explaining All Three-Alternative Voting Outcomes," Journal of Economic Theory, Elsevier, vol. 87(2), pages 313-355, August.
    11. Debopam Bhattacharya, 2015. "Nonparametric Welfare Analysis for Discrete Choice," Econometrica, Econometric Society, vol. 83, pages 617-649, March.
    12. Pavlides, Marios G. & Perlman, Michael D., 2009. "How Likely Is Simpson’s Paradox?," The American Statistician, American Statistical Association, vol. 63(3), pages 226-233.
    13. Eric Gautier & Yuichi Kitamura, 2013. "Nonparametric Estimation in Random Coefficients Binary Choice Models," Econometrica, Econometric Society, vol. 81(2), pages 581-607, March.
    14. Thomas Hammond, 2007. "Rank injustice?: How the scoring method for cross-country running competitions violates major social choice principles," Public Choice, Springer, vol. 133(3), pages 359-375, December.
    15. Sven Stringer & Naomi R Wray & René S Kahn & Eske M Derks, 2011. "Underestimated Effect Sizes in GWAS: Fundamental Limitations of Single SNP Analysis for Dichotomous Phenotypes," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    16. Koessler, Frédéric & Noussair, Charles & Ziegelmeyer, Anthony, 2012. "Information aggregation and belief elicitation in experimental parimutuel betting markets," Journal of Economic Behavior & Organization, Elsevier, vol. 83(2), pages 195-208.
    17. Hausman, Jerry A & Newey, Whitney K, 1995. "Nonparametric Estimation of Exact Consumers Surplus and Deadweight Loss," Econometrica, Econometric Society, vol. 63(6), pages 1445-1476, November.
    18. Akritas, Michael G. & Antoniou, Efi S. & Kuha, Jouni, 2006. "Nonparametric Analysis of Factorial Designs With Random Missingness: Bivariate Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1513-1526, December.
    19. Miller, Joshua Benjamin & Sanjurjo, Adam, 2018. "Surprised by the Hot Hand Fallacy? A Truth in the Law of Small Numbers," OSF Preprints sv9x2, Center for Open Science.
    20. Bargagliotti, Anna E., 2009. "Aggregation and decision making using ranked data," Mathematical Social Sciences, Elsevier, vol. 58(3), pages 354-366, November.
    21. Hanson, Robin & Oprea, Ryan & Porter, David, 2006. "Information aggregation and manipulation in an experimental market," Journal of Economic Behavior & Organization, Elsevier, vol. 60(4), pages 449-459, August.
    22. Jan De Neve & Olivier Thas, 2015. "A Regression Framework for Rank Tests Based on the Probabilistic Index Model," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 1276-1283, September.
    23. Jiangtao Gou & Fengqing (Zoe) Zhang, 2017. "Experience Simpson's Paradox in the Classroom," The American Statistician, Taylor & Francis Journals, vol. 71(1), pages 61-66, January.
    24. Briesch, Richard A. & Chintagunta, Pradeep K. & Matzkin, Rosa L., 2010. "Nonparametric Discrete Choice Models With Unobserved Heterogeneity," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(2), pages 291-307.
    25. James Boudreau & Justin Ehrlich & Mian Farrukh Raza & Shane Sanders, 2018. "The likelihood of social choice violations in rank sum scoring: algorithms and evidence from NCAA cross country running," Public Choice, Springer, vol. 174(3), pages 219-238, March.
    26. Axelrod, Boris S. & Kulick, Ben J. & Plott, Charles R. & Roust, Kevin A., 2009. "The design of improved parimutuel-type information aggregation mechanisms: Inaccuracies and the long-shot bias as disequilibrium phenomena," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 170-181, February.
    27. William Gehrlein & Florenz Plassmann, 2014. "A comparison of theoretical and empirical evaluations of the Borda Compromise," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 43(3), pages 747-772, October.
    28. Kaufmann, Christine & Weber, Martin, 2013. "Sometimes less is more – The influence of information aggregation on investment decisions," Journal of Economic Behavior & Organization, Elsevier, vol. 95(C), pages 20-33.
    29. Deanna B. Haunsperger, 2003. "Aggregated statistical rankings are arbitrary," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 20(2), pages 261-272, March.
    30. Joshua B. Miller & Adam Sanjurjo, 2018. "Surprised by the Hot Hand Fallacy? A Truth in the Law of Small Numbers," Econometrica, Econometric Society, vol. 86(6), pages 2019-2047, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matzkin, Rosa L., 2019. "Constructive identification in some nonseparable discrete choice models," Journal of Econometrics, Elsevier, vol. 211(1), pages 83-103.
    2. Giovanni Compiani & Yuichi Kitamura, 2016. "Using mixtures in econometric models: a brief review and some new results," Econometrics Journal, Royal Economic Society, vol. 19(3), pages 95-127, October.
    3. Steven T. Berry & Philip A. Haile, 2009. "Identification of a Heterogeneous Generalized Regression Model with Group Effects," Cowles Foundation Discussion Papers 1732, Cowles Foundation for Research in Economics, Yale University.
    4. Steven T. Berry & Philip A. Haile, 2009. "Nonparametric Identification of Multinomial Choice Demand Models with Heterogeneous Consumers," NBER Working Papers 15276, National Bureau of Economic Research, Inc.
    5. Sebastiaan Maes & Raghav Malhotra, 2023. "Robust Hicksian Welfare Analysis under Individual Heterogeneity," Papers 2303.01231, arXiv.org, revised Nov 2023.
    6. Matzkin, Rosa L., 2012. "Identification in nonparametric limited dependent variable models with simultaneity and unobserved heterogeneity," Journal of Econometrics, Elsevier, vol. 166(1), pages 106-115.
    7. Berube, Sarah & Crisman, Karl-Dieter, 2011. "Decomposition behavior in aggregated data sets," Mathematical Social Sciences, Elsevier, vol. 61(1), pages 12-19, January.
    8. Steven T. Berry & Philip A. Haile, 2014. "Identification in Differentiated Products Markets Using Market Level Data," Econometrica, Econometric Society, vol. 82, pages 1749-1797, September.
    9. Fox, Jeremy T. & Kim, Kyoo il & Ryan, Stephen P. & Bajari, Patrick, 2012. "The random coefficients logit model is identified," Journal of Econometrics, Elsevier, vol. 166(2), pages 204-212.
    10. Martin O'Connell & Pierre Dubois & Rachel Griffith, 2022. "The Use of Scanner Data for Economics Research," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 723-745, August.
    11. Miller, Joshua B. & Sanjurjo, Adam, 2021. "Is it a fallacy to believe in the hot hand in the NBA three-point contest?," European Economic Review, Elsevier, vol. 138(C).
    12. Robert Lantis & Erik Nesson, 2021. "Hot Shots: An Analysis of the “Hot Hand†in NBA Field Goal and Free Throw Shooting," Journal of Sports Economics, , vol. 22(6), pages 639-677, August.
    13. Arthur Lewbel, 2012. "An Overview of the Special Regressor Method," Boston College Working Papers in Economics 810, Boston College Department of Economics.
    14. Patrick Bajari & Jeremy T. Fox & Kyoo il Kim & Stephen P. Ryan, 2009. "A Simple Nonparametric Estimator for the Distribution of Random Coefficients," NBER Working Papers 15210, National Bureau of Economic Research, Inc.
    15. David M. Ritzwoller & Joseph P. Romano, 2019. "Uncertainty in the Hot Hand Fallacy: Detecting Streaky Alternatives to Random Bernoulli Sequences," Papers 1908.01406, arXiv.org, revised Apr 2021.
    16. Gan, Li & Ju, Gaosheng & Zhu, Xi, 2015. "Nonparametric estimation of structural labor supply and exact welfare change under nonconvex piecewise-linear budget sets," Journal of Econometrics, Elsevier, vol. 188(2), pages 526-544.
    17. Andrew Chesher & Adam M. Rosen, 2014. "An instrumental variable random‐coefficients model for binary outcomes," Econometrics Journal, Royal Economic Society, vol. 17(2), pages 1-19, June.
    18. Roy Allen & John Rehbeck, 2020. "Identification of Random Coefficient Latent Utility Models," Papers 2003.00276, arXiv.org.
    19. Philip A. Haile & Ali Hortaçsu & Grigory Kosenok, 2008. "On the Empirical Content of Quantal Response Equilibrium," American Economic Review, American Economic Association, vol. 98(1), pages 180-200, March.
    20. Rachel Griffith & Lars Nesheim & Martin O'Connell, 2018. "Income effects and the welfare consequences of tax in differentiated product oligopoly," Quantitative Economics, Econometric Society, vol. 9(1), pages 305-341, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0228627. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.