IDEAS home Printed from https://ideas.repec.org/a/spr/jesaex/v1y2015i1d10.1007_s40881-015-0005-3.html
   My bibliography  Save this article

Online field experiments: a selective survey of methods

Author

Listed:
  • Yan Chen

    (University of Michigan)

  • Joseph Konstan

    (University of Minnesota)

Abstract

The Internet presents today’s researchers with unprecedented opportunities to conduct field experiments. Using examples from Economics and Computer Science, we present an analysis of the design choices, with particular attention to the underlying technologies, in conducting online field experiments and report on lessons learned.

Suggested Citation

  • Yan Chen & Joseph Konstan, 2015. "Online field experiments: a selective survey of methods," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 1(1), pages 29-42, July.
    • Handle: RePEc:spr:jesaex:v:1:y:2015:i:1:d:10.1007_s40881-015-0005-3
    DOI: 10.1007/s40881-015-0005-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40881-015-0005-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s40881-015-0005-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Stephen Leider & Markus M. Möbius & Tanya Rosenblat & Quoc-Anh Do, 2009. "Directed Altruism and Enforced Reciprocity in Social Networks," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(4), pages 1815-1851.
    2. repec:hal:spmain:info:hdl:2441/14otokka698nb83lk2n7bhqbo2 is not listed on IDEAS
    3. repec:feb:artefa:0106 is not listed on IDEAS
    4. Kevin J. Boudreau & Karim R. Lakhani, 2011. "The Confederacy of Heterogeneous Software Organizations and Heterogeneous Developers: Field Experimental Evidence on Sorting and Worker Effort," NBER Chapters, in: The Rate and Direction of Inventive Activity Revisited, pages 483-502, National Bureau of Economic Research, Inc.
    5. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    6. Nicola Lacetera & Mario Macis & Robert Slonim, 2012. "Will There Be Blood? Incentives and Displacement Effects in Pro-social Behavior," American Economic Journal: Economic Policy, American Economic Association, vol. 4(1), pages 186-223, February.
    7. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    8. Yan Chen & F. Maxwell Harper & Joseph Konstan & Sherry Xin Li, 2010. "Social Comparisons and Contributions to Online Communities: A Field Experiment on MovieLens," American Economic Review, American Economic Association, vol. 100(4), pages 1358-1398, September.
    9. Paul Resnick & Richard Zeckhauser & John Swanson & Kate Lockwood, 2006. "The value of reputation on eBay: A controlled experiment," Experimental Economics, Springer;Economic Science Association, vol. 9(2), pages 79-101, June.
    10. Elena Katok & Anthony Kwasnica, 2008. "Time is money: The effect of clock speed on seller’s revenue in Dutch auctions," Experimental Economics, Springer;Economic Science Association, vol. 11(4), pages 344-357, December.
    11. Chen Yan & Li Xin & MacKie-Mason Jeffrey K, 2005. "Online Fund-Raising Mechanisms: A Field Experiment," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-39, December.
    12. Laura Gee, 2014. "The More you Know: Information Effects in Job Application Rates by Gender in a Large Field Experiment," Discussion Papers Series, Department of Economics, Tufts University 0780, Department of Economics, Tufts University.
    13. David Lucking-Reiley, 1999. "Using Field Experiments to Test Equivalence between Auction Formats: Magic on the Internet," American Economic Review, American Economic Association, vol. 89(5), pages 1063-1080, December.
    14. John List, 2008. "Informed consent in social science," Artefactual Field Experiments 00086, The Field Experiments Website.
    15. Tracy Xiao Liu & Jiang Yang & Lada A. Adamic & Yan Chen, 2014. "Crowdsourcing with All-Pay Auctions: A Field Experiment on Taskcn," Management Science, INFORMS, vol. 60(8), pages 2020-2037, August.
    16. Hal R. Varian, 2014. "Big Data: New Tricks for Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 3-28, Spring.
    17. Yan Chen & Teck‐Hua Ho & Yong‐Mi Kim, 2010. "Knowledge Market Design: A Field Experiment at Google Answers," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 12(4), pages 641-664, August.
    18. Eric J. Friedman* & Paul Resnick, 2001. "The Social Cost of Cheap Pseudonyms," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 10(2), pages 173-199, June.
    19. Amanda Pallais, 2014. "Inefficient Hiring in Entry-Level Labor Markets," American Economic Review, American Economic Association, vol. 104(11), pages 3565-3599, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dieckmann, Anja & Grimm, Veronika & Unfried, Matthias & Utikal, Verena & Valmasoni, Lorenzo, 2016. "On trust in honesty and volunteering among Europeans: Cross-country evidence on perceptions and behavior," European Economic Review, Elsevier, vol. 90(C), pages 225-253.
    2. Sander Onderstal, 2020. "Premium auctions in the field," Review of Economic Design, Springer;Society for Economic Design, vol. 24(1), pages 39-63, June.
    3. Annetta Burger & Talha Oz & William G. Kennedy & Andrew T. Crooks, 2019. "Computational Social Science of Disasters: Opportunities and Challenges," Future Internet, MDPI, vol. 11(5), pages 1-31, April.
    4. Yuan Yuan & Tracy Liu & Chenhao Tan & Qian Chen & Alex Pentland & Jie Tang, 2019. "Gift Contagion in Online Groups: Evidence From Virtual Red Packets," Papers 1906.09698, arXiv.org, revised Aug 2023.
    5. Idris Adjerid & Rachael Purta & Aaron Striegel & George Loewenstein, 2018. "Aggressive Economic Incentives and Physical Activity: The Role of Choice and Technology Decision Aids," Papers 1810.06698, arXiv.org, revised Nov 2018.
    6. Gagnon, Nickolas & Bosmans, Kristof & Riedl, Arno, 2020. "The Effect of Unfair Chances and Gender Discrimination on Labor Supply," Research Memorandum 005, Maastricht University, Graduate School of Business and Economics (GSBE).
    7. Evangelos Mourelatos & Nicholas Giannakopoulos & Manolis Tzagarakis, 2022. "Personality traits and performance in online labour markets," Behaviour and Information Technology, Taylor & Francis Journals, vol. 41(3), pages 468-484, February.
    8. Konrad Grabiszewski & Alex Horenstein, 2022. "Profiling dynamic decision-makers," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-22, April.
    9. Xiu Chen & Fuhai Hong & Xiaojian Zhao, 2020. "Concentration and variability of forecasts in artificial investment games: an online experiment on WeChat," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 815-847, September.
    10. Gong, Jie & Liu, Tracy Xiao & Tang, Jie, 2021. "How monetary incentives improve outcomes in MOOCs: Evidence from a field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 905-921.
    11. Idris Adjerid & George Loewenstein & Rachael Purta & Aaron Striegel, 2022. "Gain-Loss Incentives and Physical Activity: The Role of Choice and Wearable Health Tools," Management Science, INFORMS, vol. 68(4), pages 2642-2667, April.
    12. Dieckmann, Anja & Fischbacher, Urs & Grimm, Veronika & Unfried, Matthias & Utikal, Verena & Valmasoni, Lorenzo, 2015. "Trust and beliefs among Europeans: Cross-country evidence on perceptions and behavior," FAU Discussion Papers in Economics 04/2015, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    13. Grabiszewski, Konrad & Horenstein, Alex, 2022. "Measuring tree complexity with response times," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bolton, Gary E. & Ockenfels, Axel, 2014. "Does laboratory trading mirror behavior in real world markets? Fair bargaining and competitive bidding on eBay," Journal of Economic Behavior & Organization, Elsevier, vol. 97(C), pages 143-154.
    2. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    3. Mingfeng Lin & Yong Liu & Siva Viswanathan, 2018. "Effectiveness of Reputation in Contracting for Customized Production: Evidence from Online Labor Markets," Management Science, INFORMS, vol. 64(1), pages 345-359, January.
    4. Katkar Rama & Reiley David H, 2007. "Public versus Secret Reserve Prices in eBay Auctions: Results from a Pokémon Field Experiment," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-25, January.
    5. Anya Savikhin & Roman Sheremeta, 2010. "Visibility of Contributions and Cost of Information: An Experiment on Public Goods," Working Papers 10-22, Chapman University, Economic Science Institute.
    6. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    7. Paul Resnick & Richard Zeckhauser & John Swanson & Kate Lockwood, 2006. "The value of reputation on eBay: A controlled experiment," Experimental Economics, Springer;Economic Science Association, vol. 9(2), pages 79-101, June.
    8. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    9. Nicola Lacetera & Mario Macis & Robert Slonim, 2014. "Rewarding Volunteers: A Field Experiment," Management Science, INFORMS, vol. 60(5), pages 1107-1129, May.
    10. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    11. List John A., 2007. "Field Experiments: A Bridge between Lab and Naturally Occurring Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-47, April.
    12. Lingfang (Ivy) Li & Erte Xiao, 2014. "Money Talks: Rebate Mechanisms in Reputation System Design," Management Science, INFORMS, vol. 60(8), pages 2054-2072, August.
    13. Anya Savikhin Samek & Roman Sheremeta, 2014. "Recognizing contributors: an experiment on public goods," Experimental Economics, Springer;Economic Science Association, vol. 17(4), pages 673-690, December.
    14. John J. Horton & Richard J. Zeckhauser, 2016. "The Causes of Peer Effects in Production: Evidence from a Series of Field Experiments," NBER Working Papers 22386, National Bureau of Economic Research, Inc.
    15. Marios Kokkodis & Panagiotis G. Ipeirotis, 2016. "Reputation Transferability in Online Labor Markets," Management Science, INFORMS, vol. 62(6), pages 1687-1706, June.
    16. Bolton, Gary E. & Ockenfels, Axel, 2012. "Behavioral economic engineering," Journal of Economic Psychology, Elsevier, vol. 33(3), pages 665-676.
    17. Luís Cabral & Lingfang (Ivy) Li, 2015. "A Dollar for Your Thoughts: Feedback-Conditional Rebates on eBay," Management Science, INFORMS, vol. 61(9), pages 2052-2063, September.
    18. Anya Samek & Roman Sheremeta, 2013. "Recognizing Contributors and Cost of Information: An Experiment on Public Goods," Artefactual Field Experiments 00430, The Field Experiments Website.
    19. Antonio Moreno & Christian Terwiesch, 2014. "Doing Business with Strangers: Reputation in Online Service Marketplaces," Information Systems Research, INFORMS, vol. 25(4), pages 865-886, December.
    20. Fabo, B., 2017. "Towards an understanding of job matching using web data," Other publications TiSEM b8b877f2-ae6a-495f-b6cc-9, Tilburg University, School of Economics and Management.

    More about this item

    Keywords

    Online field experiment; A/B testing; Internet;
    All these keywords.

    JEL classification:

    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • H41 - Public Economics - - Publicly Provided Goods - - - Public Goods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jesaex:v:1:y:2015:i:1:d:10.1007_s40881-015-0005-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.