Advanced Search
MyIDEAS: Login to save this paper or follow this series

Mechanism Design for the truthful elicitation of costly probabilistic estimates in Distributed Information Systems

Contents:

Author Info

  • Papakonstantinou, A.
  • Rogers, A
  • Gerding, E. H.
  • Jennings, N. R.

Abstract

This paper reports on the design of a novel two-stage mechanism, based on strictly proper scoring rules, that allows a centre to acquire a costly forecast of a future event (such as a meteorological phenomenon) or a probabilistic estimate of a specific parameter (such as the quality of an expected service), with a specified minimum precision, from one or more agents. In the first stage, the centre elicits the agents' true costs and identifies the agent that can provide an estimate of the specified precision at the lowest cost. Then, in the second stage, the centre uses an appropriately scaled strictly proper scoring rule to incentivise this agent to generate the estimate with the required precision, and to truthfully report it. In particular, this is the first mechanism that can be applied to settings in which the centre has no knowledge about the actual costs involved in the generation an agents' estimates and also has no external means of evaluating the quality and accuracy of the estimates it receives. En route to this mechanism, we first consider a setting in which any single agent can provide an estimate of the required precision, and the centre can evaluate this estimate by comparing it with the outcome which is observed at a later stage. This mechanism is then extended, so that it can be applied in a setting where the agents' different capabilities are reflected in the maximum precision of the estimates that they can provide, potentially requiring the centre to select multiple agents and combine their individual results in order to obtain an estimate of the required precision. For all three mechanisms (the original and the two extensions), we prove their economic properties (i.e. incentive compatibility and individual rationality) and then perform a number of numerical simulations. For the single agent mechanism we compare the quadratic, spherical and logarithmic scoring rules with a parametric family of scoring rules. We show that although the logarithmic scoring rule minimises both the mean and variance of the centre's total payments, using this rule means that an agent may face an unbounded penalty if it provides an estimate of extremely poor quality. We show that this is not the case for the parametric family, and thus, we suggest that the parametric scoring rule is the best candidate in our setting. Furthermore, we show that the 'multiple agent' extension describes a family of possible approaches to select agents in the first stage of our mechanism, and we show empirically and prove analytically that there is one approach that dominates all others. Finally, we compare our mechanism to the peer prediction mechanism introduced by Miller et al. (2007) [29] and show that the centre's total expected payment is the same in both mechanisms (and is equal to total expected payment in the case that the estimates can be compared to the actual outcome), while the variance in these payments is significantly reduced within our mechanism.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://mpra.ub.uni-muenchen.de/43324/
File Function: original version
Download Restriction: no

Bibliographic Info

Paper provided by University Library of Munich, Germany in its series MPRA Paper with number 43324.

as in new window
Length:
Date of creation: Oct 2010
Date of revision:
Publication status: Published in Artificial Intelligence 2.175(2011): pp. 648-672
Handle: RePEc:pra:mprapa:43324

Contact details of provider:
Postal: Schackstr. 4, D-80539 Munich, Germany
Phone: +49-(0)89-2180-2219
Fax: +49-(0)89-2180-3900
Web page: http://mpra.ub.uni-muenchen.de
More information through EDIRC

Related research

Keywords: Multiagent systems; Scoring rules; Auction theory; Mechanism design;

Find related papers by JEL classification:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Philippe Jehiel & Benny Moldovanu, 1998. "Efficient Design with Interdependent Valuations," Discussion Papers, Northwestern University, Center for Mathematical Studies in Economics and Management Science 1244, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
  2. Papakonstantinou, A. & Rogers, A. & Gerding, E. H. & Jennings, N. R, 2008. "A Truthful Two-Stage Mechanism for Eliciting Probabilistic Estimates with Unknown Costs," MPRA Paper 43320, University Library of Munich, Germany.
  3. Claudio Mezzetti, 2005. "Mechanism Design with Interdependent Valuations: Surplus Extraction," Discussion Papers in Economics, Department of Economics, University of Leicester 05/1, Department of Economics, University of Leicester, revised Mar 2006.
  4. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Discussion Papers, Stanford Institute for Economic Policy Research 03-025, Stanford Institute for Economic Policy Research.
  5. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, University of Buckingham Press, vol. 1(1), pages 3-15, February.
  6. Selten, Reinhard, 1996. "Axiomatic Characterization of the Quadratic Scoring Rule," Discussion Paper Serie B, University of Bonn, Germany 390, University of Bonn, Germany.
  7. Yokoo, Makoto & Sakurai, Yuko & Matsubara, Shigeo, 2004. "The effect of false-name bids in combinatorial auctions: new fraud in internet auctions," Games and Economic Behavior, Elsevier, Elsevier, vol. 46(1), pages 174-188, January.
  8. Groves, Theodore, 1973. "Incentives in Teams," Econometrica, Econometric Society, Econometric Society, vol. 41(4), pages 617-31, July.
  9. Robert Day & Paul Milgrom, 2008. "Core-selecting package auctions," International Journal of Game Theory, Springer, Springer, vol. 36(3), pages 393-407, March.
  10. William Vickrey, 1961. "Counterspeculation, Auctions, And Competitive Sealed Tenders," Journal of Finance, American Finance Association, American Finance Association, vol. 16(1), pages 8-37, 03.
  11. repec:reg:wpaper:259 is not listed on IDEAS
  12. Grossman, Sanford J & Hart, Oliver D, 1983. "An Analysis of the Principal-Agent Problem," Econometrica, Econometric Society, Econometric Society, vol. 51(1), pages 7-45, January.
  13. Papakonstantinou, A. & Rogers, A. & Gerding, E. H & Jennings, N. R., 2010. "Mechanism Design for eliciting probabilistic estimates from multiple suppliers with unknown costs and limited precision," MPRA Paper 43323, University Library of Munich, Germany.
  14. Mas-Colell, Andreu & Whinston, Michael D. & Green, Jerry R., 1995. "Microeconomic Theory," OUP Catalogue, Oxford University Press, Oxford University Press, number 9780195102680, October.
  15. Claudio Mezzetti, 2004. "Mechanism Design with Interdependent Valuations: Efficiency," Econometrica, Econometric Society, Econometric Society, vol. 72(5), pages 1617-1626, 09.
  16. repec:reg:rpubli:259 is not listed on IDEAS
  17. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, INFORMS, vol. 22(10), pages 1087-1096, June.
Full references (including those not matched with items on IDEAS)

Citations

Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Papakonstantinou, A. & Bogetoft, P., 2013. "Crowd-sourcing with uncertain quality - an auction approach," MPRA Paper 44236, University Library of Munich, Germany.

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:43324. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Ekkehart Schlicht).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.