IDEAS home Printed from https://ideas.repec.org/a/kap/theord/v88y2020i4d10.1007_s11238-019-09738-z.html
   My bibliography  Save this article

Supergrading: how diverse standards can improve collective performance in ranking tasks

Author

Listed:
  • Michael Morreau

    (UiT-The Arctic University of Norway)

Abstract

The method of supergrading is introduced for deriving a ranking of items from scores or grades awarded by several people. Individual inputs may come in different languages of grades. Diversity in grading standards is an advantage, enabling rankings derived by this method to separate more items from one another. A framework is introduced for studying grading on the basis of observations. Measures of accuracy, reliability and discrimination are developed within this framework. Ability in grading is characterized for individuals and groups as the capacity to grade reliably, accurately and at a high level of discrimination. It is shown that the collective ability of a supergrading group with diverse standards can be greater than that of a less diverse group whose members have greater ability.

Suggested Citation

  • Michael Morreau, 2020. "Supergrading: how diverse standards can improve collective performance in ranking tasks," Theory and Decision, Springer, vol. 88(4), pages 541-565, May.
    • Handle: RePEc:kap:theord:v:88:y:2020:i:4:d:10.1007_s11238-019-09738-z
    DOI: 10.1007/s11238-019-09738-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11238-019-09738-z
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11238-019-09738-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hong, Lu & Page, Scott, 2009. "Interpreted and generated signals," Journal of Economic Theory, Elsevier, vol. 144(5), pages 2174-2196, September.
    2. David V. Budescu & Han-Hui Por & Stephen B. Broomell & Michael Smithson, 2014. "The interpretation of IPCC probabilistic statements around the world," Nature Climate Change, Nature, vol. 4(6), pages 508-512, June.
    3. Michael Mastrandrea & Katharine Mach & Gian-Kasper Plattner & Ottmar Edenhofer & Thomas Stocker & Christopher Field & Kristie Ebi & Patrick Matschoss, 2011. "The IPCC AR5 guidance note on consistent treatment of uncertainties: a common approach across the working groups," Climatic Change, Springer, vol. 108(4), pages 675-691, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:cup:judgdm:v:16:y:2021:i:2:p:363-393 is not listed on IDEAS
    2. A. Kause & W. Bruine de Bruin & J. Persson & H. Thorén & L. Olsson & A. Wallin & S. Dessai & N. Vareman, 2022. "Confidence levels and likelihood terms in IPCC reports: a survey of experts from different scientific disciplines," Climatic Change, Springer, vol. 173(1), pages 1-18, July.
    3. David R. Mandel & Daniel Irwin, 2021. "Facilitating sender-receiver agreement in communicated probabilities: Is it best to use words, numbers or both?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 16(2), pages 363-393, March.
    4. Casey Helgeson & Richard Bradley & Brian Hill, 2018. "Combining probability with qualitative degree-of-certainty metrics in assessment," Climatic Change, Springer, vol. 149(3), pages 517-525, August.
    5. Marie Juanchich & Theodore G. Shepherd & Miroslav Sirota, 2020. "Negations in uncertainty lexicon affect attention, decision-making and trust," Climatic Change, Springer, vol. 162(3), pages 1677-1698, October.
    6. David Klenert & Franziska Funke & Linus Mattauch & Brian O’Callaghan, 2020. "Five Lessons from COVID-19 for Advancing Climate Change Mitigation," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 76(4), pages 751-778, August.
    7. Nicolas Roux & Joel Sobel, 2015. "Group Polarization in a Model of Information Aggregation," American Economic Journal: Microeconomics, American Economic Association, vol. 7(4), pages 202-232, November.
    8. Tomich, Thomas P. & Lidder, Preetmoninder & Coley, Mariah & Gollin, Douglas & Meinzen-Dick, Ruth & Webb, Patrick & Carberry, Peter, 2019. "Food and agricultural innovation pathways for prosperity," Agricultural Systems, Elsevier, vol. 172(C), pages 1-15.
    9. P A Hancock & William G Volante, 2020. "Quantifying the qualities of language," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-27, May.
    10. Carl-Friedrich Schleussner & Joeri Rogelj & Michiel Schaeffer & Tabea Lissner & Rachel Licker & Erich M. Fischer & Reto Knutti & Anders Levermann & Katja Frieler & William Hare, 2016. "Science and policy characteristics of the Paris Agreement temperature goal," Nature Climate Change, Nature, vol. 6(9), pages 827-835, September.
    11. Ottmar Edenhofer & Carlo Carraro & Jean-Charles Hourcade, 2012. "On the economics of decarbonization in an imperfect world," Climatic Change, Springer, vol. 114(1), pages 1-8, September.
    12. Feliciani, Thomas & Morreau, Michael & Luo, Junwen & Lucas, Pablo & Shankar, Kalpana, 2022. "Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model," Research Policy, Elsevier, vol. 51(4).
    13. Michael D. Gerst & Melissa A. Kenney & Irina Feygina, 2021. "Improving the usability of climate indicator visualizations through diagnostic design principles," Climatic Change, Springer, vol. 166(3), pages 1-22, June.
    14. Mike Farjam & Olexandr Nikolaychuk & Giangiacomo Bravo, 2019. "Investing into climate change mitigation despite the risk of failure," Climatic Change, Springer, vol. 154(3), pages 453-460, June.
    15. Torres, Cati & Faccioli, Michela & Riera Font, Antoni, 2017. "Waiting or acting now? The effect on willingness-to-pay of delivering inherent uncertainty information in choice experiments," Ecological Economics, Elsevier, vol. 131(C), pages 231-240.
    16. repec:but:manage:v:4:y:2014:i:1:p:151-163 is not listed on IDEAS
    17. van der Bles, Anne Marthe & van der Liden, Sander & Freeman, Alessandra L. J. & Mitchell, James & Galvao, Ana Beatriz & Spiegelhalter, David J., 2019. "Communicating uncertainty about facts, numbers, and science," EMF Research Papers 22, Economic Modelling and Forecasting Group.
    18. Shu Liu & Yong Wang & Guang J. Zhang & Linyi Wei & Bin Wang & Le Yu, 2022. "Contrasting influences of biogeophysical and biogeochemical impacts of historical land use on global economic inequality," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    19. Lydia Gorn & Janina Kleemann & Christine Fürst, 2018. "Improving the Matrix-Assessment of Ecosystem Services Provision—The Case of Regional Land Use Planning under Climate Change in the Region of Halle, Germany," Land, MDPI, vol. 7(2), pages 1-18, June.
    20. Richard S. J. Tol, 2016. "The Impacts Of Climate Change According To The Ipcc," Climate Change Economics (CCE), World Scientific Publishing Co. Pte. Ltd., vol. 7(01), pages 1-20, February.
    21. Marina Baldissera Pacchetti & Suraje Dessai & David A. Stainforth & Seamus Bradley, 2021. "Assessing the quality of state-of-the-art regional climate information: the case of the UK Climate Projections 2018," Climatic Change, Springer, vol. 168(1), pages 1-25, September.
    22. Marie Juanchich & Miroslav Sirota, 2016. "How to improve people's interpretation of probabilities of precipitation," Journal of Risk Research, Taylor & Francis Journals, vol. 19(3), pages 388-404, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:theord:v:88:y:2020:i:4:d:10.1007_s11238-019-09738-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.