IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11503.html
   My bibliography  Save this paper

A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator

Author

Listed:
  • Smyk, Magdalena

    () (University of Warsaw)

  • Tyrowicz, Joanna

    () (University of Warsaw)

  • van der Velde, Lucas

    () (GRAPE)

Abstract

We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60% of the analyzed 95 datasets systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persists even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Suggested Citation

  • Smyk, Magdalena & Tyrowicz, Joanna & van der Velde, Lucas, 2018. "A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator," IZA Discussion Papers 11503, Institute for the Study of Labor (IZA).
  • Handle: RePEc:iza:izadps:dp11503
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp11503.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Rajeev Dehejia & Adriana Lleras-Muney, 2004. "Booms, Busts, and Babies' Health," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 1091-1130.
    2. James, M.J., 2008. "The digital divide across all citizens of the world : A new concept," Other publications TiSEM 83add553-11e1-4e2c-83b8-9, Tilburg University, School of Economics and Management.
    3. Sloczynski, Tymon, 2015. "Average Wage Gaps and Oaxaca–Blinder Decompositions," IZA Discussion Papers 9036, Institute for the Study of Labor (IZA).
    4. Jérôme Adda & Christian Dustmann & Katrien Stevens, 2017. "The Career Costs of Children," Journal of Political Economy, University of Chicago Press, vol. 125(2), pages 293-337.
    5. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    6. Stefan Bauernschuster & Timo Hener & Helmut Rainer, 2016. "Children Of A (Policy) Revolution: The Introduction Of Universal Child Care And Its Effect On Fertility," Journal of the European Economic Association, European Economic Association, vol. 14(4), pages 975-1005, August.
    7. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    8. Sarah Sinclair & Jonathan Boymal & Ashton De Silva, 2012. "A Re‐Appraisal of the Fertility Response to the Australian Baby Bonus," The Economic Record, The Economic Society of Australia, vol. 88(s1), pages 78-87, June.
    9. Cremer, Helmuth & Gahvari, Firouz & Pestieau, Pierre, 2006. "Pensions with endogenous and stochastic fertility," Journal of Public Economics, Elsevier, vol. 90(12), pages 2303-2321, December.
    10. Jeffrey James, 2008. "The Digital Divide Across All Citizens of the World: A New Concept," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(2), pages 275-282, November.
    11. repec:pri:cheawb:adriana_booms.pdf is not listed on IDEAS
    12. Heinrich Hock & David Weil, 2012. "On the dynamics of the age structure, dependency, and consumption," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(3), pages 1019-1043, July.
    13. Thomas Baudin, 2011. "Family Policies: What Does the Standard Endogenous Fertility Model Tell Us?," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 13(4), pages 555-593, August.
    14. Tijdens Kea, 2014. "Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey," Journal of Official Statistics, Sciendo, vol. 30(1), pages 23-43, March.
    15. repec:aia:aiaswp:wp102 is not listed on IDEAS
    16. Petra E. Todd & Jeffrey A. Smith, 2001. "Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods," American Economic Review, American Economic Association, vol. 91(2), pages 112-118, May.
    17. Ben Jann, 2008. "The Blinder–Oaxaca decomposition for linear regression models," Stata Journal, StataCorp LP, vol. 8(4), pages 453-479, December.
    18. Momota, Akira, 2016. "Intensive and extensive margins of fertility, capital accumulation, and economic welfare," Journal of Public Economics, Elsevier, vol. 133(C), pages 90-110.
    19. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    20. Shinichi Nishiyama & Kent Smetters, 2007. "Does Social Security Privatization Produce Efficiency Gains?," The Quarterly Journal of Economics, Oxford University Press, vol. 122(4), pages 1677-1719.
    21. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    22. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    23. Erosa, Andres & Fuster, Luisa & Restuccia, Diego, 2016. "A quantitative theory of the gender gap in wages," European Economic Review, Elsevier, vol. 85(C), pages 165-187.
    24. Martin Huber, 2011. "Testing for covariate balance using quantile regression and resampling methods," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(12), pages 2881-2899, February.
    25. Attanasio, Orazio & Levell, Peter & Low, Hamish & Sánchez-Marcos, Virginia, 2015. "Aggregating Elasticities: Intensive and Extensive Margins of Female Labour Supply," CEPR Discussion Papers 10732, C.E.P.R. Discussion Papers.
    26. Gray, Jeffrey S, 1998. "Divorce-Law Changes, Household Bargaining, and Married Women's Labor Supply," American Economic Review, American Economic Association, vol. 88(3), pages 628-642, June.
    27. John Casterline & Siqi Han, 2017. "Unrealized fertility: Fertility desires at the end of the reproductive career," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 36(14), pages 427-454, January.
    28. Maarten van Klaveren & Kea Tijdens & Stefano Visintin, 2015. "Skill Mismatch among Migrant Workers: Evidence from A Large Multi-Country Dataset," Working Papers id:7342, eSocialSciences.
    29. Claudia Olivetti & Barbara Petrongolo, 2017. "The Economic Consequences of Family Policies: Lessons from a Century of Legislation in High-Income Countries," Journal of Economic Perspectives, American Economic Association, vol. 31(1), pages 205-230, Winter.
    30. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    31. Alan S. Blinder, 1973. "Wage Discrimination: Reduced Form and Structural Estimates," Journal of Human Resources, University of Wisconsin Press, vol. 8(4), pages 436-455.
    32. Pau Baizan & Bruno Arpino & Carlos Eric Delclòs, 2016. "The Effect of Gender Policies on Fertility: The Moderating Role of Education and Normative Context," European Journal of Population, Springer;European Association for Population Studies, vol. 32(1), pages 1-30, February.
    33. Georges, Patrick & Seçkin, Aylin, 2016. "From pro-natalist rhetoric to population policies in Turkey? An OLG general equilibrium analysis," Economic Modelling, Elsevier, vol. 56(C), pages 79-93.
    34. Robert Drago & Katina Sawyer & Karina M Shreffler & Diana Warren & Mark Wooden, 2009. "Did Australia's Baby Bonus Increase the Fertility Rate?," Melbourne Institute Working Paper Series wp2009n01, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    35. Goerg, Sebastian J. & Kaiser, Johannes, 2009. "Nonparametric testing of distributions—the Epps–Singleton two-sample test using the empirical characteristic function," Stata Journal, StataCorp LP, vol. 0(Number 3), pages 1-12.
    36. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    37. Stefano Visintin & Kea Tijdens & Maarten van Klaveren, 2015. "Skill mismatch among migrant workers: evidence from a large multi-country dataset," IZA Journal of Migration and Development, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-34, December.
    38. Guzi, Martin & de Pedraza, Pablo, 2013. "A Web Survey Analysis of the Subjective Well-being of Spanish Workers," IZA Discussion Papers 7618, Institute for the Study of Labor (IZA).
    39. repec:aia:aiaswp:wp76 is not listed on IDEAS
    40. Fehr, Hans & Kallweit, Manuel & Kindermann, Fabian, 2017. "Families and social security," European Economic Review, Elsevier, vol. 91(C), pages 30-56.
    41. repec:pri:cheawb:adriana_booms is not listed on IDEAS
    42. Liao, Pei-Ju, 2011. "Does demographic change matter for growth?," European Economic Review, Elsevier, vol. 55(5), pages 659-677, June.
    43. Martin Guzi & Pablo de Pedraza García, 2015. "A web survey analysis of subjective well-being," International Journal of Manpower, Emerald Group Publishing, vol. 36(1), pages 48-67, April.
    44. Havnes, Tarjei & Mogstad, Magne, 2011. "Money for nothing? Universal child care and maternal employment," Journal of Public Economics, Elsevier, vol. 95(11), pages 1455-1465.
    45. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    46. Kevin Milligan, 2005. "Subsidizing the Stork: New Evidence on Tax Incentives and Fertility," The Review of Economics and Statistics, MIT Press, vol. 87(3), pages 539-555, August.
    47. Rafael Lalive & Josef Zweimüller, 2009. "How Does Parental Leave Affect Fertility and Return to Work? Evidence from Two Natural Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1363-1402.
    48. Tomas Frejka & Sergei Zakharov, 2013. "The Apparent Failure of Russia's Pronatalist Family Policies," Population and Development Review, The Population Council, Inc., vol. 39(4), pages 635-647, December.
    49. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    50. Rozemarijn Dereuddre & Bart Van de Putte & Piet Bracke, 2016. "Ready, Willing, and Able: Contraceptive Use Patterns Across Europe," European Journal of Population, Springer;European Association for Population Studies, vol. 32(4), pages 543-573, October.
    51. Braunsberger, Karin & Wybenga, Hans & Gates, Roger, 2007. "A comparison of reliability between telephone and web-based surveys," Journal of Business Research, Elsevier, vol. 60(7), pages 758-764, July.
    52. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    53. repec:cai:poeine:pope_1102_0361 is not listed on IDEAS
    54. Santiago Garganta & Leonardo Gasparini & Mariana Marchionni & Mariano Tappatá, 2017. "The Effect of Cash Transfers on Fertility: Evidence from Argentina," Population Research and Policy Review, Springer;Southern Demographic Association (SDA), vol. 36(1), pages 1-24, February.
    55. Guy Laroque & Bernard Salanié, 2014. "Identifying The Response Of Fertility To Financial Incentives," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(2), pages 314-332, March.
    56. Bina Agarwal, 1997. "''Bargaining'' and Gender Relations: Within and Beyond the Household," Feminist Economics, Taylor & Francis Journals, vol. 3(1), pages 1-51.
    57. Richard Valliant & Jill A. Dever, 2011. "Estimating Propensity Adjustments for Volunteer Web Surveys," Sociological Methods & Research, , vol. 40(1), pages 105-137, February.
    58. Wolfgang Lutz & Vegard Skirbekk, 2005. "Policies Addressing the Tempo Effect in Low-Fertility Countries," Population and Development Review, The Population Council, Inc., vol. 31(4), pages 699-720.
    59. Schmid, Friedrich & Trede, Mark, 1995. "A distribution free test for the two sample problem for general alternatives," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 409-419, October.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    Wage Indicator; online surveys; propensity score matching; weights;

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J30 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - General
    • J31 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - Wage Level and Structure; Wage Differentials

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11503. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.