IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v253y2017i1d10.1007_s10479-016-2341-y.html
   My bibliography  Save this article

Judgement and ranking: living with hidden bias

Author

Listed:
  • António Osório

    (Universitat Rovira i Virgili and CREIP)

Abstract

The complexity and subjectivity of the judgement task conceals the existence of biases that undermines the quality of the process. This paper presents a weighted aggregation function that attempts to reduce the influence of biased judgements on the final score. We also discuss a set of desirable properties. The proposed weighted aggregation function is able to correct the “nationalism bias” found by Emerson et al. (Am Stat 63(2):124–131, 2009) in the 2000 Olympic Games diving competition and suggest the possibility of a “reputation bias”. Our results can be applied to judgement sports and other activities that require the aggregation of several personal evaluations.

Suggested Citation

  • António Osório, 2017. "Judgement and ranking: living with hidden bias," Annals of Operations Research, Springer, vol. 253(1), pages 501-518, June.
    • Handle: RePEc:spr:annopr:v:253:y:2017:i:1:d:10.1007_s10479-016-2341-y
    DOI: 10.1007/s10479-016-2341-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-016-2341-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-016-2341-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Nolan Miller & Paul Resnick & Richard Zeckhauser, 2005. "Eliciting Informative Feedback: The Peer-Prediction Method," Management Science, INFORMS, vol. 51(9), pages 1359-1373, September.
    2. Jungmin Lee, 2008. "Outlier Aversion in Subjective Evaluation," Journal of Sports Economics, , vol. 9(2), pages 141-159, April.
    3. Victor A. Ginsburgh & Jan C. van Ours, 2003. "Expert Opinion and Compensation: Evidence from a Musical Competition," American Economic Review, American Economic Association, vol. 93(1), pages 289-296, March.
    4. repec:dau:papers:123456789/15122 is not listed on IDEAS
    5. Pavel Yu. Chebotarev & Elena Shamis, 1998. "Characterizations of scoring methodsfor preference aggregation," Annals of Operations Research, Springer, vol. 80(0), pages 299-332, January.
    6. Satterthwaite, Mark Allen, 1975. "Strategy-proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions," Journal of Economic Theory, Elsevier, vol. 10(2), pages 187-217, April.
    7. John T. Buchanan & Erez J. Henig & Mordecai I. Henig, 1998. "Objectivity and subjectivity in thedecision making process," Annals of Operations Research, Springer, vol. 80(0), pages 333-345, January.
    8. Michel Balinski & Rida Laraki, 2014. "Judge: Don't Vote !," Operations Research, INFORMS, vol. 62(3), pages 483-511, June.
    9. Eric Zitzewitz, 2006. "Nationalism in Winter Sports Judging and Its Lessons for Organizational Decision Making," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 15(1), pages 67-99, March.
    10. Michel Balinski & Rida Laraki, 2011. "Election by Majority Judgment: Experimental Evidence," Studies in Public Choice, in: Bernard Dolez & Bernard Grofman & Annie Laurent (ed.), In Situ and Laboratory Experiments on Electoral Law Reform, chapter 0, pages 13-54, Springer.
    11. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    12. Smith, John H, 1973. "Aggregation of Preferences with Variable Electorate," Econometrica, Econometric Society, vol. 41(6), pages 1027-1041, November.
    13. Emerson, John W. & Seltzer, Miki & Lin, David, 2009. "Assessing Judging Bias: An Example From the 2000 Olympic Games," The American Statistician, American Statistical Association, vol. 63(2), pages 124-131.
    14. Kenneth J. Arrow, 1950. "A Difficulty in the Concept of Social Welfare," Journal of Political Economy, University of Chicago Press, vol. 58, pages 328-328.
    15. Michel Grabisch & Jean-Luc Marichal & Radko Mesiar & Endre Pap, 2011. "Aggregation functions: construction methods, conjunctive, disjunctive and mixed classes," Post-Print hal-00539032, HAL.
    16. Gibbard, Allan, 1973. "Manipulation of Voting Schemes: A General Result," Econometrica, Econometric Society, vol. 41(4), pages 587-601, July.
    17. Hodgson, Robert T., 2008. "An Examination of Judge Reliability at a major U.S. Wine Competition," Journal of Wine Economics, Cambridge University Press, vol. 3(2), pages 105-113, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. António Osório, 2020. "Performance Evaluation: Subjectivity, Bias and Judgment Style in Sport," Group Decision and Negotiation, Springer, vol. 29(4), pages 655-678, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Osório, António (António Miguel), 2016. "Judgement and Ranking: Living with Hidden Bias," Working Papers 2072/267264, Universitat Rovira i Virgili, Department of Economics.
    2. António Osório, 2020. "Performance Evaluation: Subjectivity, Bias and Judgment Style in Sport," Group Decision and Negotiation, Springer, vol. 29(4), pages 655-678, August.
    3. L'aszl'o Csat'o, 2019. "Journal ranking should depend on the level of aggregation," Papers 1904.06300, arXiv.org, revised Sep 2019.
    4. Csató, László, 2017. "European qualifiers to the 2018 FIFA World Cup can be manipulated," MPRA Paper 82652, University Library of Munich, Germany.
    5. Michel Balinski & Rida Laraki, 2014. "Judge: Don't Vote !," Operations Research, INFORMS, vol. 62(3), pages 483-511, June.
    6. Burak Can & Peter Csoka & Emre Ergin, 2017. "How to choose a non-manipulable delegation?," CERS-IE WORKING PAPERS 1713, Institute of Economics, Centre for Economic and Regional Studies.
    7. Csóka, Péter & Kondor, Gábor, 2019. "Delegációk igazságos kiválasztása társadalmi választások elméletével [Choosing a fair delegation by social choice theory]," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(7), pages 771-787.
    8. Conal Duddy, 2014. "Condorcet’s principle and the strong no-show paradoxes," Theory and Decision, Springer, vol. 77(2), pages 275-285, August.
    9. Tomas J. McIntee, 2017. "A geometric model of sensitivity of multistage elections to change," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 49(1), pages 89-115, June.
    10. Wolitzky, Alexander, 2009. "Fully sincere voting," Games and Economic Behavior, Elsevier, vol. 67(2), pages 720-735, November.
    11. Roger B. Myerson, 1991. "Proportional Representation," Discussion Papers 928, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    12. Pierre Bernhard & Marc Deschamps, 2018. "Arrow’s (im)possibility theorem," Post-Print hal-01941037, HAL.
    13. Harless, Patrick, 2015. "Solidarity with respect to small changes in preferences in public good economies with single-peaked preferences," Mathematical Social Sciences, Elsevier, vol. 75(C), pages 81-86.
    14. Gaudeul, Alexia, 2009. "A (micro) course in microeconomic theory for MSc students," MPRA Paper 15388, University Library of Munich, Germany.
    15. Haris Aziz & Alexander Lam, 2021. "Obvious Manipulability of Voting Rules," Papers 2111.01983, arXiv.org, revised Jun 2022.
    16. Ning Yu, 2015. "A quest for fundamental theorems of social choice," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 44(3), pages 533-548, March.
    17. Stensholt, Eivind, 2020. "Anomalies of Instant Runoff Voting," Discussion Papers 2020/6, Norwegian School of Economics, Department of Business and Management Science.
    18. Kotaro Suzumura, 2002. "Introduction to social choice and welfare," Temi di discussione (Economic working papers) 442, Bank of Italy, Economic Research and International Relations Area.
    19. Gergaud, Olivier & Ginsburgh, Victor & Moreno-Ternero, Juan D., 2021. "Wine Ratings: Seeking a Consensus among Tasters via Normalization, Approval, and Aggregation," Journal of Wine Economics, Cambridge University Press, vol. 16(3), pages 321-342, August.
    20. Qiaoming Han & Donglei Du & Dachuan Xu & Yicheng Xu, 2018. "Approximate efficiency and strategy-proofness for moneyless mechanisms on single-dipped policy domain," Journal of Global Optimization, Springer, vol. 70(4), pages 859-873, April.

    More about this item

    Keywords

    Weighted aggregation function; Judgement by grading; Nationalism bias; Reputation bias; Bias correction; Olympic games;
    All these keywords.

    JEL classification:

    • D72 - Microeconomics - - Analysis of Collective Decision-Making - - - Political Processes: Rent-seeking, Lobbying, Elections, Legislatures, and Voting Behavior
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:253:y:2017:i:1:d:10.1007_s10479-016-2341-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.