IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_12535.html

The Collection of High-Quality Online Data

Author

Listed:
  • Can Celebi
  • Christine Exley
  • Soren Harrs
  • Hannu Kivimaki
  • Marta Serra-Garcia
  • Jeffrey Yusof

Abstract

Absent high-quality online data, research questions would be constrained conceptually and in study populations. To inform the debate about online data quality, this paper provides empirical evidence that compares data quality of responses from online participants, AI agents, and human subjects in the lab. Corresponding results reveal high data quality on some platforms, but not others. This paper also highlights a viable path for high-quality online data in an evolving landscape: use a two-stage recruitment method to broadly recruit online subjects in a baseline study and then limit recruitment for the main study to the resulting subset of "high quality" subjects.

Suggested Citation

  • Can Celebi & Christine Exley & Soren Harrs & Hannu Kivimaki & Marta Serra-Garcia & Jeffrey Yusof, 2026. "The Collection of High-Quality Online Data," CESifo Working Paper Series 12535, CESifo.
    • Handle: RePEc:ces:ceswps:_12535
    as

    Download full text from publisher

    File URL: https://www.ifo.de/DocDL/cesifo1_wp12535.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Stefanie Stantcheva, 2023. "How to Run Surveys: A Guide to Creating Your Own Identifying Variation and Revealing the Invisible," Annual Review of Economics, Annual Reviews, vol. 15(1), pages 205-234, September.
    2. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    3. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    4. Sahil Chinoy & Nathan Nunn & Sandra Sequeira & Stefanie Stantcheva, 2026. "Zero-Sum Thinking and the Roots of US Political Differences," American Economic Review, American Economic Association, vol. 116(3), pages 1052-1096, March.
    5. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    6. Simone Zhang & Janet Xu & AJ Alvero, 2025. "Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization," Sociological Methods & Research, , vol. 54(3), pages 1197-1242, August.
    7. Pedro Bordalo & Giovanni Burro & Katherine Coffman & Nicola Gennaioli & Andrei Shleifer, 2025. "Imagining the Future: Memory, Simulation, and Beliefs," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 92(3), pages 1532-1563.
    8. Erik Snowberg & Leeat Yariv, 2021. "Testing the Waters: Behavior across Participant Pools," American Economic Review, American Economic Association, vol. 111(2), pages 687-719, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brunner, Fabian & Gamm, Fabian & Mill, Wladislaw, 2023. "MyPortfolio: The IKEA effect in financial investment decisions," Journal of Banking & Finance, Elsevier, vol. 154(C).
    2. Bortolotti, Stefania & Kölle, Felix & Wenner, Lukas, 2022. "On the persistence of dishonesty," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 1053-1065.
    3. Ronayne, David & Sgroi, Daniel & Tuckwell, Anthony, 2021. "Evaluating the sunk cost effect," Journal of Economic Behavior & Organization, Elsevier, vol. 186(C), pages 318-327.
    4. Zhang, Yinjunjie & Hoffmann, Manuel & Sara, Raisa & Eckel, Catherine, 2024. "Fairness preferences revisited," Journal of Economic Behavior & Organization, Elsevier, vol. 223(C), pages 278-306.
    5. Bhatt, Vipul & Smith, Angela M., 2025. "Overconfidence and performance: Evidence from a simple real-effort task," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    6. Hiroki Ozono & Daisuke Nakama, 2022. "Effects of experimental situation on group cooperation and individual performance: Comparing laboratory and online experiments," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-17, April.
    7. Hyndman, Kyle & Walker, Matthew J., 2022. "Fairness and risk in ultimatum bargaining," Games and Economic Behavior, Elsevier, vol. 132(C), pages 90-105.
    8. Mariconda, Simone & Lurati, Francesco, 2015. "Does familiarity breed stability? The role of familiarity in moderating the effects of new information on reputation judgments," Journal of Business Research, Elsevier, vol. 68(5), pages 957-964.
    9. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, IZA Network @ LISER.
    10. Gökçe Esenduran & James A. Hill & In Joon Noh, 2020. "Understanding the Choice of Online Resale Channel for Used Electronics," Production and Operations Management, Production and Operations Management Society, vol. 29(5), pages 1188-1211, May.
    11. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    12. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    13. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    14. Efrat Dressler & Yevgeny Mugerman, 2023. "Doing the Right Thing? The Voting Power Effect and Institutional Shareholder Voting," Journal of Business Ethics, Springer, vol. 183(4), pages 1089-1112, April.
    15. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    16. Olenka Kacperczyk & Peter Younkin & Vera Rocha, 2023. "Do Employees Work Less for Female Leaders? A Multi-Method Study of Entrepreneurial Firms," Organization Science, INFORMS, vol. 34(3), pages 1111-1133, May.
    17. Gupta, Vishal K. & Goktan, A. Banu & Gunay, Gonca, 2014. "Gender differences in evaluation of new business opportunity: A stereotype threat perspective," Journal of Business Venturing, Elsevier, vol. 29(2), pages 273-288.
    18. Garbarino, Ellen & Slonim, Robert & Villeval, Marie Claire, 2019. "Loss aversion and lying behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 158(C), pages 379-393.
    19. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    20. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_12535. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.