IDEAS home Printed from https://ideas.repec.org/a/kap/expeco/v24y2021i3d10.1007_s10683-020-09680-w.html
   My bibliography  Save this article

Optimal design of experiments to identify latent behavioral types

Author

Listed:
  • Stefano Balietti

    (Mannheim Center for European Social Science Research (MZES), Mannheim University
    Heidelberg University)

  • Brennan Klein

    (Northeastern University)

  • Christoph Riedl

    (Northeastern University
    Northeastern University
    Northeastern University
    Harvard University)

Abstract

Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a “wisdom of experts” prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.

Suggested Citation

  • Stefano Balietti & Brennan Klein & Christoph Riedl, 2021. "Optimal design of experiments to identify latent behavioral types," Experimental Economics, Springer;Economic Science Association, vol. 24(3), pages 772-799, September.
  • Handle: RePEc:kap:expeco:v:24:y:2021:i:3:d:10.1007_s10683-020-09680-w
    DOI: 10.1007/s10683-020-09680-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10683-020-09680-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10683-020-09680-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sarin, Rajiv & Vahid, Farshid, 2001. "Predicting How People Play Games: A Simple Dynamic Model of Choice," Games and Economic Behavior, Elsevier, vol. 34(1), pages 104-122, January.
    2. Eric M. Schwartz & Eric T. Bradlow & Peter S. Fader, 2017. "Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments," Marketing Science, INFORMS, vol. 36(4), pages 500-522, July.
    3. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    4. Ron Berman, 2018. "Beyond the Last Touch: Attribution in Online Advertising," Marketing Science, INFORMS, vol. 37(5), pages 771-792, September.
    5. El-Gamal, Mahmoud A & Palfrey, Thomas R, 1996. "Economical Experiments: Bayesian Efficient Experimental Design," International Journal of Game Theory, Springer;Game Theory Society, vol. 25(4), pages 495-517.
    6. Teck H. Ho & Xin Wang & Colin F. Camerer, 2008. "Individual Differences in EWA Learning with Partial Payoff Information," Economic Journal, Royal Economic Society, vol. 118(525), pages 37-59, January.
    7. David, Paul A, 1985. "Clio and the Economics of QWERTY," American Economic Review, American Economic Association, vol. 75(2), pages 332-337, May.
    8. Colin Camerer & Teck-Hua Ho, 1999. "Experience-weighted Attraction Learning in Normal Form Games," Econometrica, Econometric Society, vol. 67(4), pages 827-874, July.
    9. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    10. Timothy C. Salmon, 2001. "An Evaluation of Econometric Models of Adaptive Learning," Econometrica, Econometric Society, vol. 69(6), pages 1597-1628, November.
    11. McIntyre, David P. & Chintakananda, Asda, 2014. "Competing in network markets: Can the winner take all?," Business Horizons, Elsevier, vol. 57(1), pages 117-125.
    12. John C. Harsanyi, 1967. "Games with Incomplete Information Played by "Bayesian" Players, I-III Part I. The Basic Model," Management Science, INFORMS, vol. 14(3), pages 159-182, November.
    13. Katz, Michael L & Shapiro, Carl, 1985. "Network Externalities, Competition, and Compatibility," American Economic Review, American Economic Association, vol. 75(3), pages 424-440, June.
    14. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," CESifo Working Paper Series 7262, CESifo.
    15. Marc Knez & Colin Camerer, 1994. "Creating Expectational Assets in the Laboratory: Coordination in ‘Weakest‐Link’ Games," Strategic Management Journal, Wiley Blackwell, vol. 15(S1), pages 101-119, December.
    16. Gale, John & Binmore, Kenneth G. & Samuelson, Larry, 1995. "Learning to be imperfect: The ultimatum game," Games and Economic Behavior, Elsevier, vol. 8(1), pages 56-90.
    17. Nick Feltovich, 2000. "Reinforcement-Based vs. Belief-Based Learning Models in Experimental Asymmetric-Information," Econometrica, Econometric Society, vol. 68(3), pages 605-642, May.
    18. TeckH. Ho & Xin Wang & ColinF. Camerer, 2008. "Individual Differences in EWA Learning with Partial Payoff Information," Economic Journal, Royal Economic Society, vol. 118(525), pages 37-59, January.
    19. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    20. Duncan Sheppard Gilchrist & Emily Glassberg Sands, 2016. "Something to Talk About: Social Spillovers in Movie Consumption," Journal of Political Economy, University of Chicago Press, vol. 124(5), pages 1339-1382.
    21. Chaim Fershtman & Ariel Pakes, 2012. "Dynamic Games with Asymmetric Information: A Framework for Empirical Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1611-1661.
    22. Erev, I. & Roth, Alvin E., 2014. "Maximization, learning, and economic behavior," Scholarly Articles 30831199, Harvard University Department of Economics.
    23. Ben M. Parker & Steven G. Gilmour & John Schormans, 2017. "Optimal design of experiments on connected units with application to social networks," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 66(3), pages 455-480, April.
    24. Sobol, I.M., 1998. "On quasi-Monte Carlo integrations," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 47(2), pages 103-112.
    25. Steven Kachelmeier & Kristy Towry, 2005. "The Limitations of Experimental Design: A Case Study Involving Monetary Incentive Effects in Laboratory Markets," Experimental Economics, Springer;Economic Science Association, vol. 8(1), pages 21-33, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Healy, Paul J. & Park, Hyoeun, 2023. "Model selection accuracy in behavioral game theory: A simulation," European Economic Review, Elsevier, vol. 152(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xie, Erhao, 2021. "Empirical properties and identification of adaptive learning models in behavioral game theory," Journal of Economic Behavior & Organization, Elsevier, vol. 191(C), pages 798-821.
    2. repec:wyi:journl:002151 is not listed on IDEAS
    3. Masiliūnas, Aidas, 2023. "Learning in rent-seeking contests with payoff risk and foregone payoff information," Games and Economic Behavior, Elsevier, vol. 140(C), pages 50-72.
    4. Shachat, Jason & Swarthout, J. Todd, 2012. "Learning about learning in games through experimental control of strategic interdependence," Journal of Economic Dynamics and Control, Elsevier, vol. 36(3), pages 383-402.
    5. Erhao Xie, 2019. "Monetary Payoff and Utility Function in Adaptive Learning Models," Staff Working Papers 19-50, Bank of Canada.
    6. Ed Hopkins, 2002. "Two Competing Models of How People Learn in Games," Econometrica, Econometric Society, vol. 70(6), pages 2141-2166, November.
    7. Asim Ansari & Ricardo Montoya & Oded Netzer, 2012. "Dynamic learning in behavioral games: A hidden Markov mixture of experts approach," Quantitative Marketing and Economics (QME), Springer, vol. 10(4), pages 475-503, December.
    8. Cooper, David J. & Stockman, Carol Kraker, 2002. "Fairness and learning: an experimental examination," Games and Economic Behavior, Elsevier, vol. 41(1), pages 26-45, October.
    9. Duffy, John, 2006. "Agent-Based Models and Human Subject Experiments," Handbook of Computational Economics, in: Leigh Tesfatsion & Kenneth L. Judd (ed.), Handbook of Computational Economics, edition 1, volume 2, chapter 19, pages 949-1011, Elsevier.
    10. Camerer, Colin F. & Ho, Teck-Hua, 2015. "Behavioral Game Theory Experiments and Modeling," Handbook of Game Theory with Economic Applications,, Elsevier.
    11. Wen, Yuanji, 2018. "Voluntary information acquisition in an asymmetric-Information game:comparing learning theories in the laboratory," Journal of Economic Behavior & Organization, Elsevier, vol. 150(C), pages 202-219.
    12. Bigoni, Maria & Fort, Margherita, 2013. "Information and learning in oligopoly: An experiment," Games and Economic Behavior, Elsevier, vol. 81(C), pages 192-214.
    13. Breitmoser, Yves, 2019. "Knowing me, imagining you: Projection and overbidding in auctions," Games and Economic Behavior, Elsevier, vol. 113(C), pages 423-447.
    14. Wolf Ze'ev Ehrblatt & Kyle Hyndman & Erkut Y. ÄOzbay & Andrew Schotter, 2006. "Convergence: An Experimental Study," Levine's Working Paper Archive 122247000000001148, David K. Levine.
    15. Wu, Hang & Bayer, Ralph-C, 2015. "Learning from inferred foregone payoffs," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 445-458.
    16. Ralph-C. Bayer & Elke Renner & Rupert Sausgruber, 2013. "Confusion and learning in the voluntary contributions game," Experimental Economics, Springer;Economic Science Association, vol. 16(4), pages 478-496, December.
    17. Breitmoser, Yves, 2017. "Knowing Me, Imagining You:," Rationality and Competition Discussion Paper Series 36, CRC TRR 190 Rationality and Competition.
    18. Ulrich Doraszelski & Gregory Lewis & Ariel Pakes, 2018. "Just Starting Out: Learning and Equilibrium in a New Market," American Economic Review, American Economic Association, vol. 108(3), pages 565-615, March.
    19. David Cooper & Nick Feltovich & Alvin Roth & Rami Zwick, 2003. "Relative versus Absolute Speed of Adjustment in Strategic Environments: Responder Behavior in Ultimatum Games," Experimental Economics, Springer;Economic Science Association, vol. 6(2), pages 181-207, October.
    20. Kawagoe, Toshiji & Takizawa, Hirokazu, 2012. "Level-k analysis of experimental centipede games," Journal of Economic Behavior & Organization, Elsevier, vol. 82(2), pages 548-566.
    21. Michael P. Haselhuhn & Devin G. Pope & Maurice E. Schweitzer & Peter Fishman, 2012. "The Impact of Personal Experience on Behavior: Evidence from Video-Rental Fines," Management Science, INFORMS, vol. 58(1), pages 52-61, January.

    More about this item

    Keywords

    Optimal experimental design; Behavioral types; Expert prediction; Active learning;
    All these keywords.

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C72 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - Noncooperative Games

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:expeco:v:24:y:2021:i:3:d:10.1007_s10683-020-09680-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.