IDEAS home Printed from https://ideas.repec.org/a/kap/theord/v85y2018i1d10.1007_s11238-017-9643-7.html
   My bibliography  Save this article

The wisdom of collective grading and the effects of epistemic and semantic diversity

Author

Listed:
  • Aidan Lyon

    (University of Maryland at College Park)

  • Michael Morreau

    (UiT-The Arctic University of Norway)

Abstract

A computer simulation is used to study collective judgements that an expert panel reaches on the basis of qualitative probability judgements contributed by individual members. The simulated panel displays a strong and robust crowd wisdom effect. The panel’s performance is better when members contribute precise probability estimates instead of qualitative judgements, but not by much. Surprisingly, it does not always hurt for panel members to interpret the probability expressions differently. Indeed, coordinating their understandings can be much worse.

Suggested Citation

  • Aidan Lyon & Michael Morreau, 2018. "The wisdom of collective grading and the effects of epistemic and semantic diversity," Theory and Decision, Springer, vol. 85(1), pages 99-116, July.
    • Handle: RePEc:kap:theord:v:85:y:2018:i:1:d:10.1007_s11238-017-9643-7
    DOI: 10.1007/s11238-017-9643-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11238-017-9643-7
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11238-017-9643-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Claus Beisbart & Stephan Hartmann, 2010. "Welfarist evaluations of decision rules under interstate utility dependencies," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 34(2), pages 315-344, February.
    2. Claus Beisbart & Luc Bovens, 2007. "Welfarist evaluations of decision rules for boards of representatives," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 29(4), pages 581-608, December.
    3. José Alcantud & Annick Laruelle, 2014. "Dis&approval voting: a characterization," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 43(1), pages 1-10, June.
    4. Marcus Pivato, 2016. "Asymptotic utilitarianism in scoring rules," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 47(2), pages 431-458, August.
    5. Gaertner, Wulf & Xu, Yongsheng, 2012. "A general scoring rule," Mathematical Social Sciences, Elsevier, vol. 63(3), pages 193-196.
    6. Claus Beisbart & Luc Bovens & Stephan Hartmann, 2005. "A Utilitarian Assessment of Alternative Decision Rules in the Council of Ministers," European Union Politics, , vol. 6(4), pages 395-418, December.
    7. Aleskerov, Fuad & Yakuba, Vyacheslav & Yuzbashev, Dmitriy, 2007. "A `threshold aggregation' of three-graded rankings," Mathematical Social Sciences, Elsevier, vol. 53(1), pages 106-110, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas Feliciani & Ramanathan Moorthy & Pablo Lucas & Kalpana Shankar, 2020. "Grade Language Heterogeneity in Simulation Models of Peer Review," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 23(3), pages 1-8.
    2. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    3. Feliciani, Thomas & Morreau, Michael & Luo, Junwen & Lucas, Pablo & Shankar, Kalpana, 2022. "Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model," Research Policy, Elsevier, vol. 51(4).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marcus Pivato, 2016. "Asymptotic utilitarianism in scoring rules," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 47(2), pages 431-458, August.
    2. Le Breton, Michel & Montero, Maria & Zaporozhets, Vera, 2012. "Voting power in the EU council of ministers and fair decision making in distributive politics," Mathematical Social Sciences, Elsevier, vol. 63(2), pages 159-173.
    3. Erdamar, Bora & Sanver, M. Remzi & Sato, Shin, 2017. "Evaluationwise strategy-proofness," Games and Economic Behavior, Elsevier, vol. 106(C), pages 227-238.
    4. Antoinette Baujard & Herrade Igersheim & Isabelle Lebon, 2020. "Some regrettable grading scale effects under different versions of evaluative voting," Working Papers halshs-02926780, HAL.
    5. Stéphane Gonzalez & Annick Laruelle & Philippe Solal, 2019. "Dilemma with approval and disapproval votes," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 53(3), pages 497-517, October.
    6. Antoinette Baujard & Herrade Igersheim & Isabelle Lebon, 2021. "Some regrettable grading scale effects under different versions of evaluative voting," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 56(4), pages 803-834, May.
    7. Justin Kruger & M. Remzi Sanver, 2021. "An Arrovian impossibility in combining ranking and evaluation," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 57(3), pages 535-555, October.
    8. González, Stéphane & Laruelle, Annick & Solal, Philippe, 2017. "Neutral candidates in approval and disapproval vote," IKERLANAK info:eu-repo/grantAgreeme, Universidad del País Vasco - Departamento de Fundamentos del Análisis Económico I.
    9. Antonin Macé, 2017. "Voting with evaluations: characterizations of evaluative voting and range voting," Working Papers halshs-01222200, HAL.
    10. Le Breton, Michel & Lepelley, Dominique & Macé, Antonin & Merlin, Vincent, 2017. "Le mécanisme optimal de vote au sein du conseil des représentants d’un système fédéral," L'Actualité Economique, Société Canadienne de Science Economique, vol. 93(1-2), pages 203-248, Mars-Juin.
    11. Matthew Gould & Matthew D. Rablen, 2024. "Are World Leaders Loss Averse?," Working Papers 2024011, The University of Sheffield, Department of Economics.
    12. Laruelle, Annick, 2018. "Voting and expressing dissatisfaction: an experiment during the 2017 French Presidential election," IKERLANAK 25736, Universidad del País Vasco - Departamento de Fundamentos del Análisis Económico I.
    13. Kazuya Kikuchi & Yukio Koriyama, 2019. "The Winner-Take-All Dilemma," ISER Discussion Paper 1059r, Institute of Social and Economic Research, Osaka University, revised Dec 2019.
    14. Olivier Mouzon & Thibault Laurent & Michel Breton & Dominique Lepelley, 2019. "Exploring the effects of national and regional popular vote Interstate compact on a toy symmetric version of the Electoral College: an electoral engineering perspective," Public Choice, Springer, vol. 179(1), pages 51-95, April.
    15. Matthew Gould & Matthew D. Rablen, 2019. "Are World Leaders Loss Averse?," CESifo Working Paper Series 7763, CESifo.
    16. Macé, Antonin, 2018. "Voting with evaluations: Characterizations of evaluative voting and range voting," Journal of Mathematical Economics, Elsevier, vol. 79(C), pages 10-17.
    17. Michel Breton & Karine Straeten, 2015. "Influence versus utility in the evaluation of voting rules: a new look at the Penrose formula," Public Choice, Springer, vol. 165(1), pages 103-122, October.
    18. Laslier, Jean-François, 2012. "Why not proportional?," Mathematical Social Sciences, Elsevier, vol. 63(2), pages 90-93.
    19. Antonin Macé, 2015. "Voting with Evaluations: When Should We Sum? What Should We Sum?," AMSE Working Papers 1544, Aix-Marseille School of Economics, France, revised 29 Oct 2015.
    20. Annick Laruelle, 2021. "“Not This One”: Experimental Use of the Approval and Disapproval Ballot," Homo Oeconomicus: Journal of Behavioral and Institutional Economics, Springer, vol. 38(1), pages 15-28, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:theord:v:85:y:2018:i:1:d:10.1007_s11238-017-9643-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.