IDEAS home Printed from https://ideas.repec.org/a/eee/jbrese/v69y2016i8p3139-3148.html

A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples

Author

Listed:
  • Smith, Scott M.
  • Roster, Catherine A.
  • Golden, Linda L.
  • Albaum, Gerald S.

Abstract

With the exploding use of Internet surveys, research efforts and data quality are increasingly subject to the effects of respondents who do not give the required attention to survey questions and who speed through the survey, or who intentionally cheat with their answers. We investigate respondent integrity and data quality for samples drawn from a “Regular” online panel and from Amazon's MTurk. New metrics for assessing sample integrity and online data quality are introduced. Overall, MTurk respondents in both respondent groups took less time to answer questions. The non-USA MTurk group deviated most from correct answers in attention filter questions and had more duplicate IP addresses. In addition, the results from the three Internet sample sources are substantively different. The choice of an Internet survey sample vendor is critical, as it can impact sample composition, respondent integrity, data quality, data structure and substantive results.

Suggested Citation

  • Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    • Handle: RePEc:eee:jbrese:v:69:y:2016:i:8:p:3139-3148
    DOI: 10.1016/j.jbusres.2015.12.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S014829631500627X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jbusres.2015.12.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Robert Peterson & Dwight Merunka, 2014. "Convenience samples of college students and research reproducibility," Post-Print hal-01822317, HAL.
    2. Peterson, Robert A. & Merunka, Dwight R., 2014. "Convenience samples of college students and research reproducibility," Journal of Business Research, Elsevier, vol. 67(5), pages 1035-1041.
    3. Albaum, Gerald S. & Evangelista, Felicitas & Medina, Nila, 1998. "Role of Response Behavior Theory in Survey Research: A Cross-National Study," Journal of Business Research, Elsevier, vol. 42(2), pages 115-125, June.
    4. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    5. Braunsberger, Karin & Wybenga, Hans & Gates, Roger, 2007. "A comparison of reliability between telephone and web-based surveys," Journal of Business Research, Elsevier, vol. 60(7), pages 758-764, July.
    6. Lee, Hanjoon & Lindquist, Jay D. & Acito, Frank, 1997. "Managers' evaluation of research design and its impact on the use of research: An experimental approach," Journal of Business Research, Elsevier, vol. 39(3), pages 231-240, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    2. Almudena Martínez-Campillo & María del Pilar Sierra-Fernández & Yolanda Fernández-Santos, 2019. "Service-Learning for Sustainability Entrepreneurship in Rural Areas: What Is Its Global Impact on Business University Students?," Sustainability, MDPI, vol. 11(19), pages 1-21, September.
    3. Cristina Robledo-Ardila & Juan Pablo Román-Calderón, 2022. "Potential: in search for meaning, theory and avenues for future research a systematic review," Management Review Quarterly, Springer, vol. 72(1), pages 149-186, February.
    4. Höllig, Christoph E. & Tumasjan, Andranik & Welpe, Isabell M., 2020. "Individualizing gamified systems: The role of trait competitiveness and leaderboard design," Journal of Business Research, Elsevier, vol. 106(C), pages 288-303.
    5. Mohammad Nurul Alam & Osarodion Ogiemwonyi & Ibrahim. E. Hago & Noor Azlinna Azizan & Fariza Hashim & Md Sazzad Hossain, 2023. "Understanding Consumer Environmental Ethics and the Willingness to Use Green Products," SAGE Open, , vol. 13(1), pages 21582440221, January.
    6. Mireia Ercilla-Montserrat & David Sanjuan-Delmás & Esther Sanyé-Mengual & Laura Calvet-Mir & Karla Banderas & Joan Rieradevall & Xavier Gabarrell, 2019. "Analysis of the consumer’s perception of urban food products from a soilless system in rooftop greenhouses: a case study from the Mediterranean area of Barcelona (Spain)," Agriculture and Human Values, Springer;The Agriculture, Food, & Human Values Society (AFHVS), vol. 36(3), pages 375-393, September.
    7. Canova, Moara Almeida & Lapola, David M. & Pinho, Patrícia & Dick, Jan & Patricio, Gleiciani B. & Priess, Joerg A., 2019. "Different ecosystem services, same (dis)satisfaction with compensation: A critical comparison between farmers’ perception in Scotland and Brazil," Ecosystem Services, Elsevier, vol. 35(C), pages 164-172.
    8. Brian ‘t Hart & Ian Phau, 2022. "Conceptualising attitudes towards brand genuinuity: scale development and validation," Journal of Brand Management, Palgrave Macmillan, vol. 29(4), pages 327-340, July.
    9. José M. Ramírez-Hurtado & Alfredo G. Hernández-Díaz & Ana D. López-Sánchez & Víctor E. Pérez-León, 2021. "Measuring Online Teaching Service Quality in Higher Education in the COVID-19 Environment," IJERPH, MDPI, vol. 18(5), pages 1-14, March.
    10. Magdalena Stefańska & Iwona Olejnik, 2024. "Retailers’ social impact perspective–discrepancies between theory and practice," DECISION: Official Journal of the Indian Institute of Management Calcutta, Springer;Indian Institute of Management Calcutta, vol. 51(3), pages 355-368, September.
    11. Dwayne D. Gremler & Yves Vaerenbergh & Elisabeth C. Brüggen & Kevin P. Gwinner, 2020. "Understanding and managing customer relational benefits in services: a meta-analysis," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 565-583, May.
    12. Aysun Ata Aktürk & Hasibe Özlen Demircan, 2018. "Development of Preschool Children Sibling Rivalry Scale (PSRS)," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 11(1), pages 117-136, February.
    13. Boman, John H. & Mowen, Thomas J., 2018. "Same feathers, different flocks: Breaking down the meaning of ‘behavioral Homophily’ in the etiology of crime," Journal of Criminal Justice, Elsevier, vol. 54(C), pages 30-40.
    14. Joana César Machado & Beatriz Fonseca & Carla Martins, 2021. "Brand logo and brand gender: examining the effects of natural logo designs and color on brand gender perceptions and affect," Journal of Brand Management, Palgrave Macmillan, vol. 28(2), pages 152-170, March.
    15. Lucinda Austin & Barbara Miller Gaither, 2019. "Redefining fit: examining CSR company-issue fit in stigmatized industries," Journal of Brand Management, Palgrave Macmillan, vol. 26(1), pages 9-20, January.
    16. Grigorios, Lamprinakos & Magrizos, Solon & Kostopoulos, Ioannis & Drossos, Dimitrios & Santos, David, 2022. "Overt and covert customer data collection in online personalized advertising: The role of user emotions," Journal of Business Research, Elsevier, vol. 141(C), pages 308-320.
    17. Rašković, Matevž & Ding, Zhonghui & Škare, Vatroslav & Ozretić Došen, Đurđana & Žabkar, Vesna, 2016. "Comparing consumer innovativeness and ethnocentrism of young-adult consumers," Journal of Business Research, Elsevier, vol. 69(9), pages 3682-3686.
    18. Teng, Weichen & Su, Yaohua & Liao, Tien-Tien & Wei, Cang-Liang, 2020. "An exploration of celebrity business ventures and their appeal to fans and non-fans," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).
    19. Seok Chan Jeong & Beom-Jin Choi, 2022. "Moderating Effects of Consumers’ Personal Innovativeness on the Adoption and Purchase Intention of Wearable Devices," SAGE Open, , vol. 12(4), pages 21582440221, November.
    20. Šerić, Maja & Ozretić-Došen, Đurđana & Škare, Vatroslav, 2020. "How can perceived consistency in marketing communications influence customer–brand relationship outcomes?," European Management Journal, Elsevier, vol. 38(2), pages 335-343.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jbrese:v:69:y:2016:i:8:p:3139-3148. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jbusres .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.