IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v50y2021i1p429-464.html
   My bibliography  Save this article

A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator

Author

Listed:
  • Magdalena Smyk
  • Joanna Tyrowicz
  • Lucas van der Velde

Abstract

We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60 percent of the analyzed 95 data sets, systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persist even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Suggested Citation

  • Magdalena Smyk & Joanna Tyrowicz & Lucas van der Velde, 2021. "A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator," Sociological Methods & Research, , vol. 50(1), pages 429-464, February.
  • Handle: RePEc:sae:somere:v:50:y:2021:i:1:p:429-464
    DOI: 10.1177/0049124118782538
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124118782538
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0049124118782538?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Rajeev Dehejia & Adriana Lleras-Muney, 2004. "Booms, Busts, and Babies' Health," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 1091-1130.
    2. James, M.J., 2008. "The digital divide across all citizens of the world : A new concept," Other publications TiSEM 83add553-11e1-4e2c-83b8-9, Tilburg University, School of Economics and Management.
    3. Malhotra, Neil & Krosnick, Jon A., 2007. "The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples," Political Analysis, Cambridge University Press, vol. 15(3), pages 286-323, July.
    4. Sloczynski, Tymon, 2015. "Average Wage Gaps and Oaxaca–Blinder Decompositions," IZA Discussion Papers 9036, Institute of Labor Economics (IZA).
    5. Jérôme Adda & Christian Dustmann & Katrien Stevens, 2017. "The Career Costs of Children," Journal of Political Economy, University of Chicago Press, vol. 125(2), pages 293-337.
    6. Claudia Olivetti & Barbara Petrongolo, 2017. "The Economic Consequences of Family Policies: Lessons from a Century of Legislation in High-Income Countries," Journal of Economic Perspectives, American Economic Association, vol. 31(1), pages 205-230, Winter.
    7. Mike Brewer & Anita Ratcliffe & Sarah dSmith, 2012. "Does welfare reform affect fertility? Evidence from the UK," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(1), pages 245-266, January.
    8. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    9. Stefan Bauernschuster & Timo Hener & Helmut Rainer, 2016. "Children Of A (Policy) Revolution: The Introduction Of Universal Child Care And Its Effect On Fertility," Journal of the European Economic Association, European Economic Association, vol. 14(4), pages 975-1005, August.
    10. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    11. Sarah Sinclair & Jonathan Boymal & Ashton De Silva, 2012. "A Re‐Appraisal of the Fertility Response to the Australian Baby Bonus," The Economic Record, The Economic Society of Australia, vol. 88(s1), pages 78-87, June.
    12. Cremer, Helmuth & Gahvari, Firouz & Pestieau, Pierre, 2006. "Pensions with endogenous and stochastic fertility," Journal of Public Economics, Elsevier, vol. 90(12), pages 2303-2321, December.
    13. Jeffrey James, 2008. "The Digital Divide Across All Citizens of the World: A New Concept," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(2), pages 275-282, November.
    14. repec:pri:cheawb:adriana_booms.pdf is not listed on IDEAS
    15. Heinrich Hock & David Weil, 2012. "On the dynamics of the age structure, dependency, and consumption," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(3), pages 1019-1043, July.
    16. Thomas Baudin, 2011. "Family Policies: What Does the Standard Endogenous Fertility Model Tell Us?," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 13(4), pages 555-593, August.
    17. Tijdens Kea, 2014. "Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey," Journal of Official Statistics, Sciendo, vol. 30(1), pages 23-43, March.
    18. repec:aia:aiaswp:wp102 is not listed on IDEAS
    19. Petra E. Todd & Jeffrey A. Smith, 2001. "Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods," American Economic Review, American Economic Association, vol. 91(2), pages 112-118, May.
    20. Ben Jann, 2008. "The Blinder–Oaxaca decomposition for linear regression models," Stata Journal, StataCorp LP, vol. 8(4), pages 453-479, December.
    21. John Bongaarts & Tomáš Sobotka, 2012. "A Demographic Explanation for the Recent Rise in European Fertility," Population and Development Review, The Population Council, Inc., vol. 38(1), pages 83-120, March.
    22. Momota, Akira, 2016. "Intensive and extensive margins of fertility, capital accumulation, and economic welfare," Journal of Public Economics, Elsevier, vol. 133(C), pages 90-110.
    23. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    24. Shinichi Nishiyama & Kent Smetters, 2007. "Does Social Security Privatization Produce Efficiency Gains?," The Quarterly Journal of Economics, Oxford University Press, vol. 122(4), pages 1677-1719.
    25. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    26. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    27. Daniela Del Boca & Silvia Pasqua & Chiara Pronzato, 2009. "Motherhood and market work decisions in institutional context: a European perspective," Oxford Economic Papers, Oxford University Press, vol. 61(suppl_1), pages 147-171, April.
    28. Erosa, Andres & Fuster, Luisa & Restuccia, Diego, 2016. "A quantitative theory of the gender gap in wages," European Economic Review, Elsevier, vol. 85(C), pages 165-187.
    29. Martin Huber, 2011. "Testing for covariate balance using quantile regression and resampling methods," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(12), pages 2881-2899, February.
    30. Attanasio, Orazio & Levell, Peter & Low, Hamish & Sánchez-Marcos, Virginia, 2015. "Aggregating Elasticities: Intensive and Extensive Margins of Female Labour Supply," CEPR Discussion Papers 10732, C.E.P.R. Discussion Papers.
    31. Claudia Olivetti & Barbara Petrongolo, 2017. "The Economic Consequences of Family Policies: Lessons from a Century of Legislation," Working Papers 811, Queen Mary University of London, School of Economics and Finance.
    32. Gray, Jeffrey S, 1998. "Divorce-Law Changes, Household Bargaining, and Married Women's Labor Supply," American Economic Review, American Economic Association, vol. 88(3), pages 628-642, June.
    33. John Casterline & Siqi Han, 2017. "Unrealized fertility: Fertility desires at the end of the reproductive career," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 36(14), pages 427-454.
    34. Stefano Visintin & Kea Tijdens & Maarten van Klaveren, 2015. "Skill mismatch among migrant workers: evidence from a large multi-country dataset," IZA Journal of Migration and Development, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-34, December.
    35. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    36. Alan S. Blinder, 1973. "Wage Discrimination: Reduced Form and Structural Estimates," Journal of Human Resources, University of Wisconsin Press, vol. 8(4), pages 436-455.
    37. Pau Baizan & Bruno Arpino & Carlos Eric Delclòs, 2016. "The Effect of Gender Policies on Fertility: The Moderating Role of Education and Normative Context," European Journal of Population, Springer;European Association for Population Studies, vol. 32(1), pages 1-30, February.
    38. Massimiliano Bratti & Konstantinos Tatsiramos, 2012. "The effect of delaying motherhood on the second childbirth in Europe," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(1), pages 291-321, January.
    39. Georges, Patrick & Seçkin, Aylin, 2016. "From pro-natalist rhetoric to population policies in Turkey? An OLG general equilibrium analysis," Economic Modelling, Elsevier, vol. 56(C), pages 79-93.
    40. Robert Drago & Katina Sawyer & Karina M Shreffler & Diana Warren & Mark Wooden, 2009. "Did Australia's Baby Bonus Increase the Fertility Rate?," Melbourne Institute Working Paper Series wp2009n01, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    41. Sebastian J. Goerg & Johannes Kaiser, 2009. "Nonparametric testing of distributions—the Epps–Singleton two-sample test using the empirical characteristic function," Stata Journal, StataCorp LP, vol. 9(3), pages 454-465, September.
    42. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    43. Noriyuki Takayama & Martin Werding (ed.), 2011. "Fertility and Public Policy: How to Reverse the Trend of Declining Birth Rates," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262014513, April.
    44. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    45. Guzi, Martin & de Pedraza, Pablo, 2013. "A Web Survey Analysis of the Subjective Well-being of Spanish Workers," IZA Discussion Papers 7618, Institute of Labor Economics (IZA).
    46. repec:aia:aiaswp:wp76 is not listed on IDEAS
    47. Fehr, Hans & Kallweit, Manuel & Kindermann, Fabian, 2017. "Families and social security," European Economic Review, Elsevier, vol. 91(C), pages 30-56.
    48. repec:pri:cheawb:adriana_booms is not listed on IDEAS
    49. Liao, Pei-Ju, 2011. "Does demographic change matter for growth?," European Economic Review, Elsevier, vol. 55(5), pages 659-677, June.
    50. Martin Guzi & Pablo de Pedraza García, 2015. "A web survey analysis of subjective well-being," International Journal of Manpower, Emerald Group Publishing, vol. 36(1), pages 48-67, April.
    51. Havnes, Tarjei & Mogstad, Magne, 2011. "Money for nothing? Universal child care and maternal employment," Journal of Public Economics, Elsevier, vol. 95(11), pages 1455-1465.
    52. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    53. Kevin Milligan, 2005. "Subsidizing the Stork: New Evidence on Tax Incentives and Fertility," The Review of Economics and Statistics, MIT Press, vol. 87(3), pages 539-555, August.
    54. Rafael Lalive & Josef Zweimüller, 2009. "How Does Parental Leave Affect Fertility and Return to Work? Evidence from Two Natural Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1363-1402.
    55. Tomas Frejka & Sergei Zakharov, 2013. "The Apparent Failure of Russia's Pronatalist Family Policies," Population and Development Review, The Population Council, Inc., vol. 39(4), pages 635-647, December.
    56. Rozemarijn Dereuddre & Bart Van de Putte & Piet Bracke, 2016. "Ready, Willing, and Able: Contraceptive Use Patterns Across Europe," European Journal of Population, Springer;European Association for Population Studies, vol. 32(4), pages 543-573, October.
    57. Braunsberger, Karin & Wybenga, Hans & Gates, Roger, 2007. "A comparison of reliability between telephone and web-based surveys," Journal of Business Research, Elsevier, vol. 60(7), pages 758-764, July.
    58. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    59. repec:cai:poeine:pope_1102_0361 is not listed on IDEAS
    60. Santiago Garganta & Leonardo Gasparini & Mariana Marchionni & Mariano Tappatá, 2017. "The Effect of Cash Transfers on Fertility: Evidence from Argentina," Population Research and Policy Review, Springer;Southern Demographic Association (SDA), vol. 36(1), pages 1-24, February.
    61. Guy Laroque & Bernard Salanié, 2014. "Identifying The Response Of Fertility To Financial Incentives," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(2), pages 314-332, March.
    62. Bina Agarwal, 1997. "''Bargaining'' and Gender Relations: Within and Beyond the Household," Feminist Economics, Taylor & Francis Journals, vol. 3(1), pages 1-51.
    63. Richard Valliant & Jill A. Dever, 2011. "Estimating Propensity Adjustments for Volunteer Web Surveys," Sociological Methods & Research, , vol. 40(1), pages 105-137, February.
    64. Wolfgang Lutz & Vegard Skirbekk, 2005. "Policies Addressing the Tempo Effect in Low‐Fertility Countries," Population and Development Review, The Population Council, Inc., vol. 31(4), pages 699-720, December.
    65. Schmid, Friedrich & Trede, Mark, 1995. "A distribution free test for the two sample problem for general alternatives," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 409-419, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Joanna Tyrowicz & Krzysztof Makarski & Marcin Bielecki, 2018. "Inequality in an OLG economy with heterogeneous cohorts and pension systems," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 16(4), pages 583-606, December.
    2. de Pinto Marco & Goerke Laszlo, 2019. "Efficiency Wages in Cournot-Oligopoly," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 19(4), pages 1-13, October.
    3. Anastasios Evgenidis & Apostolos Fasianos, 2021. "Unconventional Monetary Policy and Wealth Inequalities in Great Britain," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 83(1), pages 115-175, February.
    4. Brian Fabo, 2020. "The English and Russian Language Proficiency Premium in the post-Maidan Ukraine – an Analysis of Web Survey Data," Discussion Papers 57, Central European Labour Studies Institute (CELSI).
    5. Laszlo Goerke, 2020. "Horizontal FDI in a Dynamic Cournot - Oligopoly with Endogenous Entry," IAAEU Discussion Papers 202003, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    6. Rendtel, Ulrich & Alho, Juha M., 2022. "On the fade-away of an initial bias in longitudinal surveys," Discussion Papers 2022/4, Free University Berlin, School of Business & Economics.
    7. Laszlo Goerke & Michael Neugart, 2020. "Thorstein Veblen, Joan Robinson, and George Stigler (probably) never met: Social Preferences, Monopsony, and Government Intervention," IAAEU Discussion Papers 202001, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Krzysztof Makarski & Joanna Tyrowicz & Magda Malec, 2018. "Evaluating welfare and economic effects of raised fertility," GRAPE Working Papers 25, GRAPE Group for Research in Applied Economics.
    2. Krzysztof Makarski & Joanna Tyrowicz & Magda Malec, 2019. "Fiscal and Welfare Effects of Raised Fertility in Poland: Overlapping Generations Model Estimates," Population and Development Review, The Population Council, Inc., vol. 45(4), pages 795-818, December.
    3. Tymon Słoczyński, 2015. "The Oaxaca–Blinder Unexplained Component as a Treatment Effects Estimator," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 77(4), pages 588-604, August.
    4. Elmallakh, Nelly, 2021. "Fertility, Family Policy, and Labor Supply: Quasi-Experimental Evidence from France," GLO Discussion Paper Series 984, Global Labor Organization (GLO).
    5. Advani, Arun & Sloczynski, Tymon, 2013. "Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies," IZA Discussion Papers 7874, Institute of Labor Economics (IZA).
    6. Marco Caliendo & Michel Clement & Dominik Papies & Sabine Scheel-Kopeinig, 2012. "Research Note ---The Cost Impact of Spam Filters: Measuring the Effect of Information System Technologies in Organizations," Information Systems Research, INFORMS, vol. 23(3-part-2), pages 1068-1080, September.
    7. Gordey Yastrebov, 2016. "Intergenerational Social Mobility in Soviet and Post-Soviet Russia," HSE Working papers WP BRP 69/SOC/2016, National Research University Higher School of Economics.
    8. Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2017. "The finite sample performance of semi- and non-parametric estimators for treatment effects and policy evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 91-102.
    9. Feddersen, Arne & Maennig, Wolfgang, 2012. "Sectoral labour market effects of the 2006 FIFA World Cup," Labour Economics, Elsevier, vol. 19(6), pages 860-869.
    10. Hugo Bodory & Lorenzo Camponovo & Martin Huber & Michael Lechner, 2020. "The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 183-200, January.
    11. Barigozzi, Francesca & Cremer, Helmuth & Roeder, Kerstin, 2018. "Women's career choices, social norms and child care policies," Journal of Public Economics, Elsevier, vol. 168(C), pages 162-173.
    12. Sinclair, Sarah & Boymal, Jonathan & de Silva, Ashton J, 2012. "Is the fertility response to the Australian baby bonus heterogeneous across maternal age? Evidence from Victoria," MPRA Paper 42725, University Library of Munich, Germany.
    13. Luis Aranda, 2013. "Doubling Up: A Gift or a Shame? Multigenerational Households and Parental Depression of Older Europeans," Working Papers 2013:29, Department of Economics, University of Venice "Ca' Foscari", revised 2013.
    14. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute of Labor Economics (IZA).
    15. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    16. Svetlana Biryukova & Oxana Sinyavskaya & Irina Nurimanova, 2016. "Estimating effects of 2007 family policy changes on probability of second and subsequent births in Russia," HSE Working papers WP BRP 68/SOC/2016, National Research University Higher School of Economics.
    17. Holger Stichnoth, 2020. "Short-run fertility effects of parental leave benefits: evidence from a structural model," Empirical Economics, Springer, vol. 59(1), pages 143-168, July.
    18. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    19. Raute, Anna, 2019. "Can financial incentives reduce the baby gap? Evidence from a reform in maternity leave benefits," Journal of Public Economics, Elsevier, vol. 169(C), pages 203-222.
    20. Libertad González Luna & Lidia Farré, 2017. "The effects of paternity leave on fertility and labor market outcomes," Economics Working Papers 1572, Department of Economics and Business, Universitat Pompeu Fabra.

    More about this item

    Keywords

    wage indicator; online surveys; propensity score matching; weights;
    All these keywords.

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J30 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - General
    • J31 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - Wage Level and Structure; Wage Differentials

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:50:y:2021:i:1:p:429-464. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.