Eliciting Informative Feedback: The Peer-Prediction Method
Many recommendation and decision processes depend on eliciting evaluations of opportunities, products, and vendors. A scoring system is devised that induces honest reporting of feedback. Each rater merely reports a signal, and the system applies proper scoring rules to the implied posterior beliefs about another rater's report. Honest reporting proves to be a Nash equilibrium. The scoring schemes can be scaled to induce appropriate effort by raters and can be extended to handle sequential interaction and continuous signals. We also address a number of practical implementation issues that arise in settings such as academic reviewing and online recommender and reputation systems.
Volume (Year): 51 (2005)
Issue (Month): 9 (September)
|Contact details of provider:|| Postal: 7240 Parkway Drive, Suite 300, Hanover, MD 21076 USA|
Web page: http://www.informs.org/
More information through EDIRC
References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Ottaviani, Marco & Sorensen, Peter Norman, 2006.
"The strategy of professional forecasting,"
Journal of Financial Economics,
Elsevier, vol. 81(2), pages 441-466, August.
- Marco Ottaviani & Peter Norman Sorensen, 2001. "The Strategy of Professional Forecasting," Discussion Papers 01-09, University of Copenhagen. Department of Economics.
- Marco Ottaviani & Peter Norman Sørensen, 2004. "The Strategy of Professional Forecasting," FRU Working Papers 2004/05, University of Copenhagen. Department of Economics. Finance Research Unit.
- Daniel Friedman, 1983. "Effective Scoring Rules for Probabilistic Forecasts," Management Science, INFORMS, vol. 29(4), pages 447-454, April.
- Paul Resnick & Christopher Avery & Richard Zeckhauser, 1999. "The Market for Evaluations," American Economic Review, American Economic Association, vol. 89(3), pages 564-584, June.
- R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
- Robert Clemen, 2002. "Incentive contrats and strictly proper scoring rules," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 11(1), pages 167-189, June.
- repec:kap:expeco:v:1:y:1998:i:1:p:43-62 is not listed on IDEAS
- Robert F. Nau, 1985. "Should Scoring Rules be "Effective"?," Management Science, INFORMS, vol. 31(5), pages 527-535, May.
- Cremer, Jacques & McLean, Richard P, 1985. "Optimal Selling Strategies under Uncertainty for a Discriminating Monopolist When Demands Are Interdependent," Econometrica, Econometric Society, vol. 53(2), pages 345-361, March.
- Reinhard Selten, 1998. "Axiomatic Characterization of the Quadratic Scoring Rule," Experimental Economics, Springer;Economic Science Association, vol. 1(1), pages 43-61, June.
- Selten, Reinhard, 1996. "Axiomatic Characterization of the Quadratic Scoring Rule," Discussion Paper Serie B 390, University of Bonn, Germany.
- Michihiro Kandori & Hitoshi Matsushima, 1998. "Private Observation, Communication and Collusion," Econometrica, Econometric Society, vol. 66(3), pages 627-652, May.
- Johnson, Scott & Miller, Nolan & Pratt, John W. & Zeckhauser, Richard, 2003. "Efficient Design with Multidimensional, Continuous Types, and Interdependent Valuations," Working Paper Series rwp03-020, Harvard University, John F. Kennedy School of Government.
- Cremer, Jacques & McLean, Richard P, 1988. "Full Extraction of the Surplus in Bayesian and Dominant Strategy Auctions," Econometrica, Econometric Society, vol. 56(6), pages 1247-1257, November.
- Robert G. Nelson & David A. Bessler, 1989. "Subjective Probabilities and Scoring Rules: Experimental Evidence," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 71(2), pages 363-369.
- Prendergast, Canice, 1993. "A Theory of "Yes Men."," American Economic Review, American Economic Association, vol. 83(4), pages 757-770, September.
- Johnson, Scott & Pratt, John W & Zeckhauser, Richard J, 1990. "Efficiency Despite Mutually Payoff-Relevant Private Information: The Finite Case," Econometrica, Econometric Society, vol. 58(4), pages 873-900, July. Full references (including those not matched with items on IDEAS)