IDEAS home Printed from https://ideas.repec.org/a/spr/grdene/v29y2020i4d10.1007_s10726-020-09672-4.html
   My bibliography  Save this article

Performance Evaluation: Subjectivity, Bias and Judgment Style in Sport

Author

Listed:
  • António Osório

    (Universitat Rovira i Virgili)

Abstract

The number of situations that require individual judgments and evaluations, and that may be object of different sources of conscious and unconscious biases is endless. This paper proposes a practical score aggregation procedure that attempts to reduce and mitigate the influence of bias in subjective judgments. The argument is based on the idea that bias is associated with deviations from the panel mean and/or deviations from the judges’ grading style. Consequently, the procedure is not specific to a particular type of bias, but rather addresses general forms of bias. We also discuss a set of desirable properties. The proposed score aggregation procedure is then applied to a unique data set from the 2000 Summer Olympic Games diving competition.

Suggested Citation

  • António Osório, 2020. "Performance Evaluation: Subjectivity, Bias and Judgment Style in Sport," Group Decision and Negotiation, Springer, vol. 29(4), pages 655-678, August.
    • Handle: RePEc:spr:grdene:v:29:y:2020:i:4:d:10.1007_s10726-020-09672-4
    DOI: 10.1007/s10726-020-09672-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10726-020-09672-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10726-020-09672-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sutter, Matthias & Kocher, Martin G., 2004. "Favoritism of agents - The case of referees' home bias," Journal of Economic Psychology, Elsevier, vol. 25(4), pages 461-469, August.
    2. Yates, Judith, 1985. "Discrimination in Lending," Working Papers 80, University of Sydney, School of Economics.
    3. Christopher A. Parsons & Johan Sulaeman & Michael C. Yates & Daniel S. Hamermesh, 2011. "Strike Three: Discrimination, Incentives, and Evaluation," American Economic Review, American Economic Association, vol. 101(4), pages 1410-1435, June.
    4. Joseph Price & Justin Wolfers, 2010. "Racial Discrimination Among NBA Referees," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(4), pages 1859-1887.
    5. Victor A. Ginsburgh & Jan C. van Ours, 2003. "Expert Opinion and Compensation: Evidence from a Musical Competition," American Economic Review, American Economic Association, vol. 93(1), pages 289-296, March.
    6. Ginsburgh, Victor & Noury, Abdul G., 2008. "The Eurovision Song Contest. Is voting political or cultural?," European Journal of Political Economy, Elsevier, vol. 24(1), pages 41-52, March.
    7. repec:cup:judgdm:v:3:y:2008:i::p:5-17 is not listed on IDEAS
    8. Baker, George P, 1992. "Incentive Contracts and Performance Measurement," Journal of Political Economy, University of Chicago Press, vol. 100(3), pages 598-614, June.
    9. Bruno S. Frey, 2017. "Omnimetrics and Awards," CREMA Working Paper Series 2017-09, Center for Research in Economics, Management and the Arts (CREMA).
    10. Larsen Tim & Price Joe & Wolfers Justin, 2008. "Racial Bias in the NBA: Implications in Betting Markets," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 4(2), pages 1-21, April.
    11. Eric Zitzewitz, 2006. "Nationalism in Winter Sports Judging and Its Lessons for Organizational Decision Making," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 15(1), pages 67-99, March.
    12. Daniel S. Hamermesh & Peter Schmidt, 2003. "The Determinants of Econometric Society Fellows Elections," Econometrica, Econometric Society, vol. 71(1), pages 399-407, January.
    13. Herbert A. Simon, 1955. "A Behavioral Model of Rational Choice," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 69(1), pages 99-118.
    14. Tom Coupe & Olivier Gergaud & Abdul Noury, 2018. "Biases and Strategic Behaviour in Performance Evaluation: The Case of the FIFA's best soccer player award," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 80(2), pages 358-379, April.
    15. Luis Garicano & Ignacio Palacios-Huerta & Canice Prendergast, 2005. "Favoritism Under Social Pressure," The Review of Economics and Statistics, MIT Press, vol. 87(2), pages 208-216, May.
    16. Emerson, John W. & Seltzer, Miki & Lin, David, 2009. "Assessing Judging Bias: An Example From the 2000 Olympic Games," The American Statistician, American Statistical Association, vol. 63(2), pages 124-131.
    17. Kenneth J. Arrow, 1950. "A Difficulty in the Concept of Social Welfare," Journal of Political Economy, University of Chicago Press, vol. 58, pages 328-328.
    18. Hodgson, Robert T., 2008. "An Examination of Judge Reliability at a major U.S. Wine Competition," Journal of Wine Economics, Cambridge University Press, vol. 3(2), pages 105-113, January.
    19. Jungmin Lee, 2008. "Outlier Aversion in Subjective Evaluation," Journal of Sports Economics, , vol. 9(2), pages 141-159, April.
    20. Eric Zitzewitz, 2014. "Does Transparency Reduce Favoritism and Corruption? Evidence From the Reform of Figure Skating Judging," Journal of Sports Economics, , vol. 15(1), pages 3-30, February.
    21. Satterthwaite, Mark Allen, 1975. "Strategy-proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions," Journal of Economic Theory, Elsevier, vol. 10(2), pages 187-217, April.
    22. António Osório, 2017. "Judgement and ranking: living with hidden bias," Annals of Operations Research, Springer, vol. 253(1), pages 501-518, June.
    23. Joseph Price & Marc Remer & Daniel F. Stone, 2012. "Subperfect Game: Profitable Biases of NBA Referees," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 21(1), pages 271-300, March.
    24. Justin Wolfers, 2006. "Point Shaving: Corruption in NCAA Basketball," American Economic Review, American Economic Association, vol. 96(2), pages 279-283, May.
    25. Bruno S. Frey & Jana Gallus, 2017. "Towards An Economics Of Awards," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 190-200, February.
    26. Gibbard, Allan, 1973. "Manipulation of Voting Schemes: A General Result," Econometrica, Econometric Society, vol. 41(4), pages 587-601, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ryan Rodenberg, 2013. "Employee Discipline And Basketball Referees: A Prediction Market Approach," Journal of Prediction Markets, University of Buckingham Press, vol. 7(2), pages 43-54.
    2. António Osório, 2017. "Judgement and ranking: living with hidden bias," Annals of Operations Research, Springer, vol. 253(1), pages 501-518, June.
    3. Osório, António (António Miguel), 2016. "Judgement and Ranking: Living with Hidden Bias," Working Papers 2072/267264, Universitat Rovira i Virgili, Department of Economics.
    4. Paul Gift & Ryan M. Rodenberg, 2014. "Napoleon Complex," Journal of Sports Economics, , vol. 15(5), pages 541-558, October.
    5. Thomas Dohmen & Jan Sauermann, 2016. "Referee Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 30(4), pages 679-695, September.
    6. Devin G. Pope & Joseph Price & Justin Wolfers, 2018. "Awareness Reduces Racial Bias," Management Science, INFORMS, vol. 64(11), pages 4988-4995, November.
    7. Eric Zitzewitz, 2014. "Does Transparency Reduce Favoritism and Corruption? Evidence From the Reform of Figure Skating Judging," Journal of Sports Economics, , vol. 15(1), pages 3-30, February.
    8. Paul Bose & Eberhard Feess & Helge Mueller, 2022. "Favoritism towards High-Status Clubs: Evidence from German Soccer," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 38(2), pages 422-478.
    9. Hillary N. Morgan & Kurt W. Rotthoff, 2014. "The Harder The Task, The Higher The Score: Findings Of A Difficulty Bias," Economic Inquiry, Western Economic Association International, vol. 52(3), pages 1014-1026, July.
    10. Sandra Schneemann & Hendrik Scholten & Christian Deutscher, 2018. "The Impact of Age on Nationality Bias: Evidence from Ski Jumping," Papers 1808.03804, arXiv.org.
    11. Rhett Brymer & Ryan M. Rodenberg & Huimiao Zheng & Tim R. Holcomb, 2021. "College Football Referee Bias and Sports Betting Impact," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 47(1), pages 91-106, January.
    12. Alex Krumer & Felix Otto & Tim Pawlowski, 2022. "Nationalistic bias among international experts: evidence from professional ski jumping," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(1), pages 278-300, January.
    13. Dmitry Dagaev & Sofia Paklina & J. James Reade & Carl Singleton, 2024. "The Iron Curtain and Referee Bias in International Football," Journal of Sports Economics, , vol. 25(1), pages 126-151, January.
    14. Bar-Eli, Michael & Krumer, Alex & Morgulev, Elia, 2020. "Ask not what economics can do for sports - Ask what sports can do for economics," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 89(C).
    15. Stefano Cabras & J. D. Tena, 2023. "Implicit institutional incentives and individual decisions: Causal inference with deep learning models," Managerial and Decision Economics, John Wiley & Sons, Ltd., vol. 44(6), pages 3739-3754, September.
    16. Principe, Francesco & van Ours, Jan C., 2022. "Racial bias in newspaper ratings of professional football players," European Economic Review, Elsevier, vol. 141(C).
    17. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    18. Scott Tainsky & Brian M. Mills & Jason A. Winfree, 2015. "Further Examination of Potential Discrimination Among MLB Umpires," Journal of Sports Economics, , vol. 16(4), pages 353-374, May.
    19. Brian M. Mills, 2017. "Policy Changes In Major League Baseball: Improved Agent Behavior And Ancillary Productivity Outcomes," Economic Inquiry, Western Economic Association International, vol. 55(2), pages 1104-1118, April.
    20. Bryson R. Pope & Nolan G. Pope, 2015. "Own-Nationality Bias: Evidence From Uefa Champions League Football Referees," Economic Inquiry, Western Economic Association International, vol. 53(2), pages 1292-1304, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:grdene:v:29:y:2020:i:4:d:10.1007_s10726-020-09672-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.