IDEAS home Printed from https://ideas.repec.org/p/ran/wpaper/wr-279.html
   My bibliography  Save this paper

Selection Bias in Web Surveys and the Use of Propensity Scores

Author

Listed:
  • Matthias Schonlau
  • Arthur Van Soest
  • Arie Kapteyn
  • Mick Couper

Abstract

Web surveys have several advantages compared to more traditional surveys with in-person interviews, telephone interviews, or mail surveys. Their most obvious potential drawback is that they may not be representative of the population of interest because the sub-population with access to Internet is quite specific. This paper investigates propensity scores as a method for dealing with selection bias in web surveys. The authors’ main example has an unusually rich sampling design, where the Internet sample is drawn from an existing much larger probability sample that is representative of the US 50+ population and their spouses (the Health and Retirement Study). They use this to estimate propensity scores and to construct weights based on the propensity scores to correct for selectivity. They investigate whether propensity weights constructed on the basis of a relatively small set of variables are sufficient to correct the distribution of other variables so that these distributions become representative of the population. If this is the case, information about these other variables could be collected over the Internet only. Using a backward stepwise regression they find that at a minimum all demographic variables are needed to construct the weights. The propensity adjustment works well for many but not all variables investigated. For example, they find that correcting on the basis of socio-economic status by using education level and personal income is not enough to get a representative estimate of stock ownership. This casts some doubt on the common procedure to use a few basic variables to blindly correct for selectivity in convenience samples drawn over the Internet. Alternatives include providing non-Internet users with access to the Web or conducting web surveys in the context of mixed mode surveys.

Suggested Citation

  • Matthias Schonlau & Arthur Van Soest & Arie Kapteyn & Mick Couper, 2006. "Selection Bias in Web Surveys and the Use of Propensity Scores," Working Papers WR-279, RAND Corporation.
  • Handle: RePEc:ran:wpaper:wr-279
    as

    Download full text from publisher

    File URL: https://www.rand.org/content/dam/rand/pubs/working_papers/2006/RAND_WR279.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Berrens, Robert P. & Bohara, Alok K. & Jenkins-Smith, Hank & Silva, Carol & Weimer, David L., 2003. "The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples," Political Analysis, Cambridge University Press, vol. 11(1), pages 1-22, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Elisabeth Grewenig & Philipp Lergetporer & Lisa Simon & Katharina Werner & Ludger Wößmann & Ludger Woessmann, 2018. "Can Online Surveys Represent the Entire Population?," CESifo Working Paper Series 7222, CESifo Group Munich.
    2. Joanna Tyrowicz & Magdalena Smyk & Lucas van der Velde, 2018. "A cautionary note on the reliability of the online survey data – the case of Wage Indicator," IAAEU Discussion Papers 201805, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    3. Fiore, M. & Gaviglio, A. & Demartini, E. & La Sala, P., 2018. "Sugarcoating Food Technologies and consumers’ acceptance of long-life fish," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 275971, International Association of Agricultural Economists.
    4. Arthur van Soest & Arie Kapteyn, 2009. "Mode and Context Effects in Measuring Household Assets," Working Papers 200949, Geary Institute, University College Dublin.
    5. Jeffrey R. Brown & Arie Kapteyn & Erzo F.P. Luttmer & Olivia Mitchell, 2012. "Do Consumers Know How to Value Annuities? Complexity as a Barrier to Annuitization," Working Papers WR-924-SSA, RAND Corporation.
    6. Bruine de Bruin, Wändi & van der Klaauw, Wilbert & van Rooij, Maarten & Teppa, Federica & de Vos, Klaas, 2017. "Measuring expectations of inflation: Effects of survey mode, wording, and opportunities to revise," Journal of Economic Psychology, Elsevier, vol. 59(C), pages 45-58.
    7. Hsu, Joanne W. & McFall, Brooke H., 2015. "Mode effects in mixed-mode economic surveys: Insights from a randomized experiment," Finance and Economics Discussion Series 2015-8, Board of Governors of the Federal Reserve System (US).
    8. Samantha K. Watson & Mark Elliot, 2016. "Entropy balancing: a maximum-entropy reweighting scheme to adjust for coverage error," Quality & Quantity: International Journal of Methodology, Springer, vol. 50(4), pages 1781-1797, July.
    9. Heng Chen & Geoffrey R. Dunbar & Rallye Shen, 2017. "The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design," Staff Working Papers 17-43, Bank of Canada.
    10. Randall Alan Cantrell & Amanda Stafford, 2013. "The introduction and development of the community-flow measurement instrument," Community Development, Taylor & Francis Journals, vol. 44(3), pages 305-322, July.

    More about this item

    Keywords

    surveys; methodology; computer programs;

    JEL classification:

    • C42 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Survey Methods
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ran:wpaper:wr-279. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Benson Wong). General contact details of provider: http://edirc.repec.org/data/lpranus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.