IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v52y2023i2p879-908.html

Response Quality in Nonprobability and Probability-based Online Panels

Author

Listed:
  • Carina Cornesse
  • Annelies G. Blom

Abstract

Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based online panels with identical fieldwork periods and questionnaires in Germany. Three response quality indicators typically associated with survey satisficing are assessed: straight-lining in grid questions, item nonresponse, and midpoint selection in visual design experiments. Our results show that there is significantly more straight-lining in the nonprobability online panels than in the probability-based online panels. However, contrary to our expectations, there is no generalizable difference between nonprobability online panels and probability-based online panels with respect to item nonresponse. Finally, neither respondents in nonprobability online panels nor respondents in probability-based online panels are significantly affected by the visual design of the midpoint of the answer scale.

Suggested Citation

  • Carina Cornesse & Annelies G. Blom, 2023. "Response Quality in Nonprobability and Probability-based Online Panels," Sociological Methods & Research, , vol. 52(2), pages 879-908, May.
    • Handle: RePEc:sae:somere:v:52:y:2023:i:2:p:879-908
    DOI: 10.1177/0049124120914940
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124120914940
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124120914940?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Malhotra, Neil & Krosnick, Jon A., 2007. "The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples," Political Analysis, Cambridge University Press, vol. 15(3), pages 286-323, July.
    2. Converse, Philip E., 1974. "Comment: The Status of Nonattitudes," American Political Science Review, Cambridge University Press, vol. 68(2), pages 650-660, June.
    3. Vera Toepoel & Corrie Vis & Marcel Das & Arthur van Soest, 2009. "Design of Web Questionnaires," Sociological Methods & Research, , vol. 37(3), pages 371-392, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alecia S. Lopina & Miguela B. Napiere, 2025. "Exploring the Dynamics of Arranged Marriage in a Cultural Context," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 9(7), pages 3992-3999, July.
    2. Hinz, Thomas & Marczuk, Anna & Multrus, Frank, 2024. "Student opinions on the escalation of violence in Israel and Gaza and antisemitism at German universities," Working Papers 17, University of Konstanz, Cluster of Excellence "The Politics of Inequality. Perceptions, Participation and Policies".
    3. Smith, E. Keith & Henninger, Ella & de Lauriere, Camille Fournier & Koubi, Vally & Moolla, Raeesa & Beiser-McGrath, Liam F. & Bernauer, Thomas, 2025. "Strong demand for clean air policies in emerging markets and developing economies," OSF Preprints 92bst_v1, Center for Open Science.
    4. Maren Wright Voss & Cal J. Halvorsen & Kanchan Yadav & Stephanie M. Neidlinger & Gregory R. Wagner & Susan E. Peters, 2025. "Thriving from Work Questionnaire: Validation of a Measure of Worker Wellbeing Among Older U.S. Workers," IJERPH, MDPI, vol. 22(9), pages 1-30, September.
    5. Hinz, Thomas & Marczuk, Anna & Multrus, Frank, 2024. "Studentisches Meinungsklima zur Gewalteskalation in Israel und Gaza und Antisemitismus an deutschen Hochschulen," Working Papers 16, University of Konstanz, Cluster of Excellence "The Politics of Inequality. Perceptions, Participation and Policies".
    6. Hinz, Thomas & Marczuk, Anna & Multrus, Frank, 2025. "Antisemitismus und pro-palästinensische Proteste an deutschen Hochschulen: Befragungsergebnisse bei Studierenden und Hochschulleitungen," Working Papers 43, University of Konstanz, Cluster of Excellence "The Politics of Inequality. Perceptions, Participation and Policies".
    7. Max Philipp Jansen & Yosi Yaffe, 2025. "A Tool for Examining the Role of Social Context: Cross-National Validation of the Impostor Phenomenon Short Scale (IPSS-3)," Social Sciences, MDPI, vol. 14(10), pages 1-16, September.
    8. Luciana Taddei & Dario Germani & Nicolò Marchesini & Rocco Paolillo & Claudia Pennacchiotti & Ilaria Primerano & Michele Santurro & Loredana Cerbara, 2025. "Comparison of Online Probability Panels in Europe: New Trends and Old Challenges in the Era of Open Science," Societies, MDPI, vol. 15(8), pages 1-16, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lindhjem, Henrik & Navrud, Ståle, 2011. "Using Internet in Stated Preference Surveys: A Review and Comparison of Survey Modes," International Review of Environmental and Resource Economics, now publishers, vol. 5(4), pages 309-351, September.
    2. Catherine Chen & Bo MacInnis & Matthew Waltman & Jon A. Krosnick, 2021. "Public opinion on climate change in the USA: to what extent can it be nudged by questionnaire design features?," Climatic Change, Springer, vol. 167(3), pages 1-18, August.
    3. Lasse J. Jessen & Sebastian Koehne & Patrick Nüß & Jens Ruhose, 2024. "Socioeconomic Inequality in Life Expectancy: Perception and Policy Demand," CESifo Working Paper Series 10940, CESifo.
    4. Chatpong Tangmanee & Phattharaphong Niruttinanon, 2019. "Web Survey’s Completion Rates: Effects of Forced Responses, Question Display Styles, and Subjects’ Attitude," International Journal of Research in Business and Social Science (2147-4478), Center for the Strategic Studies in Business and Finance, vol. 8(1), pages 20-29, January.
    5. Vera Toepoel & Hannah Emerson, 2017. "Using experts’ consensus (the Delphi method) to evaluate weighting techniques in web surveys not based on probability schemes," Mathematical Population Studies, Taylor & Francis Journals, vol. 24(3), pages 161-171, July.
    6. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    7. Natalja Menold & Vera Toepoel, 2024. "Do Different Devices Perform Equally Well with Different Numbers of Scale Points and Response Formats? A test of measurement invariance and reliability," Sociological Methods & Research, , vol. 53(2), pages 898-939, May.
    8. Karytsas, Spyridon & Theodoropoulou, Helen, 2014. "Socioeconomic and demographic factors that influence publics' awareness on the different forms of renewable energy sources," Renewable Energy, Elsevier, vol. 71(C), pages 480-485.
    9. Aaron C. Sparks & Heather Hodges & Sarah Oliver & Eric R. A. N. Smith, 2020. "Confidence in Local, National, and International Scientists on Climate Change," Sustainability, MDPI, vol. 13(1), pages 1-13, December.
    10. Sakshaug Joseph W. & Wiśniowski Arkadiusz & Ruiz Diego Andres Perez & Blom Annelies G., 2019. "Supplementing Small Probability Samples with Nonprobability Samples: A Bayesian Approach," Journal of Official Statistics, Sciendo, vol. 35(3), pages 653-681, September.
    11. Carina Cornesse & Ulrich Krieger & Marie‐Lou Sohnius & Marina Fikel & Sabine Friedel & Tobias Rettig & Alexander Wenz & Sebastian Juhl & Roni Lehrer & Katja Möhring & Elias Naumann & Maximiliane Reife, 2022. "From German Internet Panel to Mannheim Corona Study: Adaptable probability‐based online panel infrastructures during the pandemic," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(3), pages 773-797, July.
    12. de Bruijne, M.A., 2015. "Designing web surveys for the multi-device internet," Other publications TiSEM 19e4d446-a62b-4a95-8691-8, Tilburg University, School of Economics and Management.
    13. Stefano Visintin & Kea Tijdens & Stephanie Steinmetz & Pablo de Pedraza, 2015. "Task implementation heterogeneity and wage dispersion," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-24, December.
    14. Tobias Gummer & Tanja Kunz, 2022. "Relying on External Information Sources When Answering Knowledge Questions in Web Surveys," Sociological Methods & Research, , vol. 51(2), pages 816-836, May.
    15. Neuert Cornelia E. & Roßmann Joss & Silber Henning, 2023. "Using Eye-Tracking Methodology to Study Grid Question Designs in Web Surveys," Journal of Official Statistics, Sciendo, vol. 39(1), pages 79-101, March.
    16. Lehdonvirta, Vili & Oksanen, Atte & Räsänen, Pekka & Blank, Grant, 2020. "Social Media, Web, and Panel Surveys: Using Non- Probability Samples in Social and Policy Research," OSF Preprints qrwg4, Center for Open Science.
    17. Richard T. Carson, 2011. "Contingent Valuation," Books, Edward Elgar Publishing, number 2489, August.
    18. Magdalena Smyk & Joanna Tyrowicz & Lucas van der Velde, 2021. "A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator," Sociological Methods & Research, , vol. 50(1), pages 429-464, February.
    19. Fabo, B., 2017. "Towards an understanding of job matching using web data," Other publications TiSEM b8b877f2-ae6a-495f-b6cc-9, Tilburg University, School of Economics and Management.
    20. Dana Garbarski & Nora Cate Schaeffer & Jennifer Dykema, 2019. "The Effects of Features of Survey Measurement on Self-Rated Health: Response Option Order and Scale Orientation," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 14(2), pages 545-560, April.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:52:y:2023:i:2:p:879-908. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.