IDEAS home Printed from https://ideas.repec.org/a/sae/simgam/v50y2019i3p302-328.html
   My bibliography  Save this article

Validity Threats in Quantitative Data Collection With Games: A Narrative Survey

Author

Listed:
  • David Gundry
  • Sebastian Deterding

Abstract

Background. Games are increasingly used to collect scientific data . Some suggest that game features like high cognitive load may limit the inferences we can draw from such data, yet no systematic overview exists of potential validity threats of game-based methods . Aim. We present a narrative survey of documented and potential threats to validity in using games for quantitative data collection . Method. We combined an unsystematic bottom-up literature review with a systematic top-down application of standard validity threat typologies to games to arrive at a systematisation of game-characteristic validity threats . Results. We identify three game characteristics that potentially impact validity: Games are complex systems , impeding the predictable control and isolation of treatments. They are rich in unwanted variance and diversity. And their social framing can differ from and interact with the framing of research studies or non-game situations they are supposed to represent. The diversity of gamers and their differences to general populations bring further complications. Discussion and Conclusions. The wealth of potential validity threats in game-based research is met by a dearth of systematic methodological studies, leading us to outline several future research directions.

Suggested Citation

  • David Gundry & Sebastian Deterding, 2019. "Validity Threats in Quantitative Data Collection With Games: A Narrative Survey," Simulation & Gaming, , vol. 50(3), pages 302-328, June.
    • Handle: RePEc:sae:simgam:v:50:y:2019:i:3:p:302-328
    DOI: 10.1177/1046878118805515
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/1046878118805515
    Download Restriction: no
    File URL: https://libkey.io/10.1177/1046878118805515?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    2. Budescu, David V, 1999. "Commentary on "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework."," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 43-45, December.
    3. Eckel, Catherine, 1999. "Commentary on "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework."," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 47-48, December.
    4. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    5. Richard N. Landers & Elena M. Auer & Andrew B. Collmus & Michael B. Armstrong, 2018. "Gamification Science, Its History and Future: Definitions and a Research Agenda," Simulation & Gaming, , vol. 49(3), pages 315-337, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J. Tuomas Harviainen, 2019. "Human-Computer Interaction and Simulation/Gaming," Simulation & Gaming, , vol. 50(3), pages 263-265, June.
    2. Kunal Rajesh Lahoti & Shivani Hanji & Pratik Kamble & Kavita Vemuri, 2023. "Impact of Loss-Framing and Risk Attitudes on Insurance Purchase: Insights from a Game-like Interface Study," Papers 2310.13300, arXiv.org.
    3. Karin Slegers & Lizzy Bleumers & Bernhard Maurer & Alina Krischkowsky & Mark Blythe, 2019. "Special Issue HCI Research Games – An Editorial," Simulation & Gaming, , vol. 50(3), pages 266-271, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben-Ner, Avner & Putterman, Louis & Kong, Fanmin & Magan, Dan, 2004. "Reciprocity in a two-part dictator game," Journal of Economic Behavior & Organization, Elsevier, vol. 53(3), pages 333-352, March.
    2. Duffy, Sean & Smith, John, 2011. "Cognitive load in the multi-player prisoner's dilemma game," MPRA Paper 30856, University Library of Munich, Germany.
    3. Chen, Chia-Ching & Chiu, I-Ming & Smith, John & Yamada, Tetsuji, 2011. "Too smart to be selfish? Measures of intelligence, social preferences, and consistency," MPRA Paper 34438, University Library of Munich, Germany.
    4. Krawczyk, Michal & Le Lec, Fabrice, 2021. "How to elicit distributional preferences: A stress-test of the equality equivalence test," Journal of Economic Behavior & Organization, Elsevier, vol. 182(C), pages 13-28.
    5. Ding, Shuze & Lugovskyy, Volodymyr & Puzzello, Daniela & Tucker, Steven & Williams, Arlington, 2018. "Cash versus extra-credit incentives in experimental asset markets," Journal of Economic Behavior & Organization, Elsevier, vol. 150(C), pages 19-27.
    6. Ondrej Rydval, 2012. "The Causal Effect of Cognitive Abilities on Economic Behavior: Evidence from a Forecasting Task with Varying Cognitive Load," Jena Economics Research Papers 2011-064, Friedrich-Schiller-University Jena.
    7. Mathieu Lefebvre & Ferdinand Vieider & Marie Villeval, 2011. "The ratio bias phenomenon: fact or artifact?," Theory and Decision, Springer, vol. 71(4), pages 615-641, October.
    8. T. Ballinger & Eric Hudson & Leonie Karkoviata & Nathaniel Wilcox, 2011. "Saving behavior and cognitive abilities," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 349-374, September.
    9. Ondrej Rydval, 2011. "The Effect of Financial Incentives and Task-specific Cognitive Abilities on Task Performance," Jena Economics Research Papers 2011-050, Friedrich-Schiller-University Jena.
    10. Karen Blumenschein & GlennC. Blomquist & Magnus Johannesson & Nancy Horn & Patricia Freeman, 2008. "Eliciting Willingness to Pay Without Bias: Evidence from a Field Experiment," Economic Journal, Royal Economic Society, vol. 118(525), pages 114-137, January.
    11. Ernest Baskin, 2018. "Increasing influenza vaccination rates via low cost messaging interventions," PLOS ONE, Public Library of Science, vol. 13(2), pages 1-9, February.
    12. Thomas Dudek & Eric R. Ulm & Ilan Noy, 2021. "Demand for Multi-Year Catastrophe Insurance Contracts: Experimental Evidence for Mitigating the Insurance Gap," CESifo Working Paper Series 9442, CESifo.
    13. Walkowitz, Gari, 2021. "Dictator game variants with probabilistic (and cost-saving) payoffs: A systematic test," Journal of Economic Psychology, Elsevier, vol. 85(C).
    14. Fredslund, Eskild Klausen & Mørkbak, Morten Raun & Gyrd-Hansen, Dorte, 2018. "Different domains – Different time preferences?," Social Science & Medicine, Elsevier, vol. 207(C), pages 97-105.
    15. Chetan Dave & Catherine Eckel & Cathleen Johnson & Christian Rojas, 2010. "Eliciting risk preferences: When is simple better?," Journal of Risk and Uncertainty, Springer, vol. 41(3), pages 219-243, December.
    16. Kai Duttle & Keigo Inukai, 2015. "Complexity Aversion: Influences of Cognitive Abilities, Culture and System of Thought," Economics Bulletin, AccessEcon, vol. 35(2), pages 846-855.
    17. Pavlo Illiashenko, 2017. "Behavioral Finance: History and Foundations," Visnyk of the National Bank of Ukraine, National Bank of Ukraine, issue 239, pages 28-54.
    18. Chen, Chia-Ching & Chiu, I-Ming & Smith, John & Yamada, Tetsuji, 2013. "Too smart to be selfish? Measures of cognitive ability, social preferences, and consistency," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 112-122.
    19. Rydval, Ondrej & Ortmann, Andreas, 2004. "How financial incentives and cognitive abilities affect task performance in laboratory settings: an illustration," Economics Letters, Elsevier, vol. 85(3), pages 315-320, December.
    20. Herrmann, Tabea & Hübler, Olaf & Menkhoff, Lukas & Schmidt, Ulrich, 2016. "Allais for the poor," Kiel Working Papers 2036, Kiel Institute for the World Economy (IfW Kiel).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:simgam:v:50:y:2019:i:3:p:302-328. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.