IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v37y2009i3p291-318.html
   My bibliography  Save this article

Selection Bias in Web Surveys and the Use of Propensity Scores

Author

Listed:
  • Matthias Schonlau

    (RAND Corporation, Pittsburgh, Pennsylvania, matt@rand.org)

  • Arthur van Soest

    (Tilburg University, Netherlands)

  • Arie Kapteyn

    (RAND Corporation, Santa Monica, California)

  • Mick Couper

    (University of Michigan, Ann Arbor)

Abstract

Web surveys are a popular survey mode, but the subpopulation with Internet access may not represent the population of interest. The authors investigate whether adjusting using weights or matching on a small set of variables makes the distributions of target variables representative of the population. This application has a rich sampling design; the Internet sample is part of an existing probability sample, the Health and Retirement Study, that is representative of the U.S. population aged 50 and older. For the dichotomous variables investigated, the adjustment helps. On average, the sample means in the Internet access sample differ by 6.5 percent before and 3.7 percent after adjustment. Still, a large number of adjusted estimates remain significantly different from their target estimates based on the complete sample. This casts doubt on the common procedure to use only a few variables to correct for the selectivity of convenience samples.

Suggested Citation

  • Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
  • Handle: RePEc:sae:somere:v:37:y:2009:i:3:p:291-318
    as

    Download full text from publisher

    File URL: http://smr.sagepub.com/content/37/3/291.abstract
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Berrens, Robert P. & Bohara, Alok K. & Jenkins-Smith, Hank & Silva, Carol & Weimer, David L., 2003. "The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples," Political Analysis, Cambridge University Press, vol. 11(1), pages 1-22, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    2. Joanna Tyrowicz & Magdalena Smyk & Lucas van der Velde, 2018. "A cautionary note on the reliability of the online survey data – the case of Wage Indicator," IAAEU Discussion Papers 201805, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    3. Fiore, M. & Gaviglio, A. & Demartini, E. & La Sala, P., 2018. "Sugarcoating Food Technologies and consumers’ acceptance of long-life fish," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 275971, International Association of Agricultural Economists.
    4. Elisabeth Grewenig & Philipp Lergetporer & Lisa Simon & Katharina Werner & Ludger Wößmann & Ludger Woessmann, 2018. "Can Online Surveys Represent the Entire Population?," CESifo Working Paper Series 7222, CESifo Group Munich.
    5. van Soest, A.H.O. & Kapteyn, A., 2009. "Mode and Context Effects of Measuring Household Assets," Discussion Paper 2009-14, Tilburg University, Center for Economic Research.
    6. Jeffrey R. Brown & Arie Kapteyn & Erzo F.P. Luttmer & Olivia Mitchell, 2012. "Do Consumers Know How to Value Annuities? Complexity as a Barrier to Annuitization," Working Papers WR-924-SSA, RAND Corporation.
    7. Bruine de Bruin, Wändi & van der Klaauw, Wilbert & van Rooij, Maarten & Teppa, Federica & de Vos, Klaas, 2017. "Measuring expectations of inflation: Effects of survey mode, wording, and opportunities to revise," Journal of Economic Psychology, Elsevier, vol. 59(C), pages 45-58.
    8. Hsu, Joanne W. & McFall, Brooke H., 2015. "Mode effects in mixed-mode economic surveys: Insights from a randomized experiment," Finance and Economics Discussion Series 2015-8, Board of Governors of the Federal Reserve System (US).
    9. Samantha K. Watson & Mark Elliot, 2016. "Entropy balancing: a maximum-entropy reweighting scheme to adjust for coverage error," Quality & Quantity: International Journal of Methodology, Springer, vol. 50(4), pages 1781-1797, July.
    10. Heng Chen & Geoffrey R. Dunbar & Rallye Shen, 2017. "The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design," Staff Working Papers 17-43, Bank of Canada.
    11. Randall Alan Cantrell & Amanda Stafford, 2013. "The introduction and development of the community-flow measurement instrument," Community Development, Taylor & Francis Journals, vol. 44(3), pages 305-322, July.

    More about this item

    Keywords

    Web surveys; selection; matching; propensity scores;

    JEL classification:

    • C42 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Survey Methods
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:37:y:2009:i:3:p:291-318. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (SAGE Publications). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.