IDEAS home Printed from https://ideas.repec.org/p/fme/wpaper/26.html
   My bibliography  Save this paper

A cautionary note on the reliability of the online survey data - the case of Wage Indicator

Author

Listed:
  • Joanna Tyrowicz

    (Group for Research in Applied Economics (GRAPE)
    University of Warsaw
    Institut für Arbeitsrecht und Arbeitsbeziehungen in der Europäischen Union (IAAEU)
    Institute of Labor Economics (IZA))

  • Magdalena Smyk

    (Group for Research in Applied Economics (GRAPE)
    Warsaw School Economics)

  • Lucas van der Velde

    (Group for Research in Applied Economics (GRAPE)
    Warsaw School Economics)

Abstract

We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60% of the analyzed 95 datasets systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persists even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Suggested Citation

  • Joanna Tyrowicz & Magdalena Smyk & Lucas van der Velde, 2018. "A cautionary note on the reliability of the online survey data - the case of Wage Indicator," GRAPE Working Papers 26, GRAPE Group for Research in Applied Economics.
  • Handle: RePEc:fme:wpaper:26
    as

    Download full text from publisher

    File URL: http://grape.org.pl/WP/26_SmykTyrowiczVanDerVelde_website.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Rajeev Dehejia & Adriana Lleras-Muney, 2004. "Booms, Busts, and Babies' Health," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 1091-1130.
    2. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    3. James, M.J., 2008. "The digital divide across all citizens of the world : A new concept," Other publications TiSEM 83add553-11e1-4e2c-83b8-9, Tilburg University, School of Economics and Management.
    4. Malhotra, Neil & Krosnick, Jon A., 2007. "The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples," Political Analysis, Cambridge University Press, vol. 15(3), pages 286-323, July.
    5. Sloczynski, Tymon, 2015. "Average Wage Gaps and Oaxaca–Blinder Decompositions," IZA Discussion Papers 9036, Institute of Labor Economics (IZA).
    6. Jérôme Adda & Christian Dustmann & Katrien Stevens, 2017. "The Career Costs of Children," Journal of Political Economy, University of Chicago Press, vol. 125(2), pages 293-337.
    7. Claudia Olivetti & Barbara Petrongolo, 2017. "The Economic Consequences of Family Policies: Lessons from a Century of Legislation in High-Income Countries," Journal of Economic Perspectives, American Economic Association, vol. 31(1), pages 205-230, Winter.
    8. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    9. Georges, Patrick & Seçkin, Aylin, 2016. "From pro-natalist rhetoric to population policies in Turkey? An OLG general equilibrium analysis," Economic Modelling, Elsevier, vol. 56(C), pages 79-93.
    10. Cremer, Helmuth & Gahvari, Firouz & Pestieau, Pierre, 2006. "Pensions with endogenous and stochastic fertility," Journal of Public Economics, Elsevier, vol. 90(12), pages 2303-2321, December.
    11. Orazio Attanasio & Peter Levell & Hamish Low & Virginia Sánchez-Marcos, 2015. "Aggregating Elasticities: Intensive and Extensive Margins of Female Labour Supply," NBER Working Papers 21315, National Bureau of Economic Research, Inc.
    12. Robert Drago & Katina Sawyer & Karina M Shreffler & Diana Warren & Mark Wooden, 2009. "Did Australia's Baby Bonus Increase the Fertility Rate?," Melbourne Institute Working Paper Series wp2009n01, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    13. Stefan Bauernschuster & Timo Hener & Helmut Rainer, 2016. "Children Of A (Policy) Revolution: The Introduction Of Universal Child Care And Its Effect On Fertility," Journal of the European Economic Association, European Economic Association, vol. 14(4), pages 975-1005, August.
    14. Sarah Sinclair & Jonathan Boymal & Ashton De Silva, 2012. "A Re‐Appraisal of the Fertility Response to the Australian Baby Bonus," The Economic Record, The Economic Society of Australia, vol. 88(s1), pages 78-87, June.
    15. Sebastian J. Goerg & Johannes Kaiser, 2009. "Nonparametric testing of distributions—the Epps–Singleton two-sample test using the empirical characteristic function," Stata Journal, StataCorp LP, vol. 9(3), pages 454-465, September.
    16. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    17. Jeffrey James, 2008. "The Digital Divide Across All Citizens of the World: A New Concept," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(2), pages 275-282, November.
    18. Stefano Visintin & Kea Tijdens & Maarten van Klaveren, 2015. "Skill mismatch among migrant workers: evidence from a large multi-country dataset," IZA Journal of Migration and Development, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-34, December.
    19. repec:pri:cheawb:adriana_booms.pdf is not listed on IDEAS
    20. Heinrich Hock & David Weil, 2012. "On the dynamics of the age structure, dependency, and consumption," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(3), pages 1019-1043, July.
    21. Thomas Baudin, 2011. "Family Policies: What Does the Standard Endogenous Fertility Model Tell Us?," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 13(4), pages 555-593, August.
    22. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    23. Erosa, Andres & Fuster, Luisa & Restuccia, Diego, 2016. "A quantitative theory of the gender gap in wages," European Economic Review, Elsevier, vol. 85(C), pages 165-187.
    24. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    25. Guzi, Martin & de Pedraza, Pablo, 2013. "A Web Survey Analysis of the Subjective Well-being of Spanish Workers," IZA Discussion Papers 7618, Institute of Labor Economics (IZA).
    26. repec:aia:aiaswp:wp76 is not listed on IDEAS
    27. Fehr, Hans & Kallweit, Manuel & Kindermann, Fabian, 2017. "Families and social security," European Economic Review, Elsevier, vol. 91(C), pages 30-56.
    28. Tijdens Kea, 2014. "Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey," Journal of Official Statistics, Sciendo, vol. 30(1), pages 23-43, March.
    29. repec:pri:cheawb:adriana_booms is not listed on IDEAS
    30. repec:aia:aiaswp:wp102 is not listed on IDEAS
    31. Petra E. Todd & Jeffrey A. Smith, 2001. "Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods," American Economic Review, American Economic Association, vol. 91(2), pages 112-118, May.
    32. Liao, Pei-Ju, 2011. "Does demographic change matter for growth?," European Economic Review, Elsevier, vol. 55(5), pages 659-677, June.
    33. Martin Guzi & Pablo de Pedraza García, 2015. "A web survey analysis of subjective well-being," International Journal of Manpower, Emerald Group Publishing, vol. 36(1), pages 48-67, April.
    34. Ben Jann, 2008. "The Blinder–Oaxaca decomposition for linear regression models," Stata Journal, StataCorp LP, vol. 8(4), pages 453-479, December.
    35. Havnes, Tarjei & Mogstad, Magne, 2011. "Money for nothing? Universal child care and maternal employment," Journal of Public Economics, Elsevier, vol. 95(11), pages 1455-1465.
    36. Momota, Akira, 2016. "Intensive and extensive margins of fertility, capital accumulation, and economic welfare," Journal of Public Economics, Elsevier, vol. 133(C), pages 90-110.
    37. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    38. Kevin Milligan, 2005. "Subsidizing the Stork: New Evidence on Tax Incentives and Fertility," The Review of Economics and Statistics, MIT Press, vol. 87(3), pages 539-555, August.
    39. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    40. Shinichi Nishiyama & Kent Smetters, 2007. "Does Social Security Privatization Produce Efficiency Gains?," The Quarterly Journal of Economics, Oxford University Press, vol. 122(4), pages 1677-1719.
    41. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    42. Alexander Ludwig & Thomas Schelkle & Edgar Vogel, 2012. "Demographic Change, Human Capital and Welfare," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 15(1), pages 94-107, January.
    43. Martin Huber, 2011. "Testing for covariate balance using quantile regression and resampling methods," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(12), pages 2881-2899, February.
    44. Rafael Lalive & Josef Zweimüller, 2009. "How Does Parental Leave Affect Fertility and Return to Work? Evidence from Two Natural Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1363-1402.
    45. Tomas Frejka & Sergei Zakharov, 2013. "The Apparent Failure of Russia's Pronatalist Family Policies," Population and Development Review, The Population Council, Inc., vol. 39(4), pages 635-647, December.
    46. Gray, Jeffrey S, 1998. "Divorce-Law Changes, Household Bargaining, and Married Women's Labor Supply," American Economic Review, American Economic Association, vol. 88(3), pages 628-642, June.
    47. John Casterline & Siqi Han, 2017. "Unrealized fertility: Fertility desires at the end of the reproductive career," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 36(14), pages 427-454.
    48. Rozemarijn Dereuddre & Bart Van de Putte & Piet Bracke, 2016. "Ready, Willing, and Able: Contraceptive Use Patterns Across Europe," European Journal of Population, Springer;European Association for Population Studies, vol. 32(4), pages 543-573, October.
    49. Maarten van Klaveren & Kea Tijdens & Stefano Visintin, 2015. "Skill Mismatch among Migrant Workers: Evidence from A Large Multi-Country Dataset," Working Papers id:7342, eSocialSciences.
    50. Braunsberger, Karin & Wybenga, Hans & Gates, Roger, 2007. "A comparison of reliability between telephone and web-based surveys," Journal of Business Research, Elsevier, vol. 60(7), pages 758-764, July.
    51. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    52. repec:cai:poeine:pope_1102_0361 is not listed on IDEAS
    53. Alan S. Blinder, 1973. "Wage Discrimination: Reduced Form and Structural Estimates," Journal of Human Resources, University of Wisconsin Press, vol. 8(4), pages 436-455.
    54. Santiago Garganta & Leonardo Gasparini & Mariana Marchionni & Mariano Tappatá, 2017. "The Effect of Cash Transfers on Fertility: Evidence from Argentina," Population Research and Policy Review, Springer;Southern Demographic Association (SDA), vol. 36(1), pages 1-24, February.
    55. Guy Laroque & Bernard Salanié, 2014. "Identifying The Response Of Fertility To Financial Incentives," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(2), pages 314-332, March.
    56. Bina Agarwal, 1997. "''Bargaining'' and Gender Relations: Within and Beyond the Household," Feminist Economics, Taylor & Francis Journals, vol. 3(1), pages 1-51.
    57. Richard Valliant & Jill A. Dever, 2011. "Estimating Propensity Adjustments for Volunteer Web Surveys," Sociological Methods & Research, , vol. 40(1), pages 105-137, February.
    58. Wolfgang Lutz & Vegard Skirbekk, 2005. "Policies Addressing the Tempo Effect in Low‐Fertility Countries," Population and Development Review, The Population Council, Inc., vol. 31(4), pages 699-720, December.
    59. Pau Baizan & Bruno Arpino & Carlos Eric Delclòs, 2016. "The Effect of Gender Policies on Fertility: The Moderating Role of Education and Normative Context," European Journal of Population, Springer;European Association for Population Studies, vol. 32(1), pages 1-30, February.
    60. Schmid, Friedrich & Trede, Mark, 1995. "A distribution free test for the two sample problem for general alternatives," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 409-419, October.
    61. Massimiliano Bratti & Konstantinos Tatsiramos, 2012. "The effect of delaying motherhood on the second childbirth in Europe," Journal of Population Economics, Springer;European Society for Population Economics, vol. 25(1), pages 291-321, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Laszlo Goerke, 2020. "Horizontal FDI in a Dynamic Cournot - Oligopoly with Endogenous Entry," IAAEU Discussion Papers 202003, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    2. Joanna Tyrowicz & Krzysztof Makarski & Marcin Bielecki, 2018. "Inequality in an OLG economy with heterogeneous cohorts and pension systems," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 16(4), pages 583-606, December.
    3. Joanna Tyrowicz & Krzysztof Makarski & Marcin Bielecki, 2018. "Inequality in an OLG economy with heterogeneous cohorts and pension systems," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 16(4), pages 583-606, December.
    4. Brian Fabo, 2020. "The English and Russian Language Proficiency Premium in the post-Maidan Ukraine – an Analysis of Web Survey Data," Discussion Papers 57, Central European Labour Studies Institute (CELSI).
    5. de Pinto Marco & Goerke Laszlo, 2019. "Efficiency Wages in Cournot-Oligopoly," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 19(4), pages 1-13, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Krzysztof Makarski & Joanna Tyrowicz & Magda Malec, 2018. "Evaluating welfare and economic effects of raised fertility," GRAPE Working Papers 25, GRAPE Group for Research in Applied Economics.
    2. Tymon Słoczyński, 2015. "The Oaxaca–Blinder Unexplained Component as a Treatment Effects Estimator," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 77(4), pages 588-604, August.
    3. Marco Caliendo & Michel Clement & Dominik Papies & Sabine Scheel-Kopeinig, 2012. "Research Note ---The Cost Impact of Spam Filters: Measuring the Effect of Information System Technologies in Organizations," Information Systems Research, INFORMS, vol. 23(3-part-2), pages 1068-1080, September.
    4. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    5. Feddersen, Arne & Maennig, Wolfgang, 2012. "Sectoral labour market effects of the 2006 FIFA World Cup," Labour Economics, Elsevier, vol. 19(6), pages 860-869.
    6. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    7. Luis Aranda, 2013. "Doubling Up: A Gift or a Shame? Multigenerational Households and Parental Depression of Older Europeans," Working Papers 2013:29, Department of Economics, University of Venice "Ca' Foscari", revised 2013.
    8. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute of Labor Economics (IZA).
    9. Del Prete, Davide & Ghins, Léopold & Magrini, Emiliano & Pauw, Karl, 2019. "Land consolidation, specialization and household diets: Evidence from Rwanda," Food Policy, Elsevier, vol. 83(C), pages 139-149.
    10. Flores, Carlos A. & Mitnik, Oscar A., 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," IZA Discussion Papers 4451, Institute of Labor Economics (IZA).
    11. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    12. Ashok K. Mishra & Anjani Kumar & Pramod K. Joshi & Alwin D'Souza, 2018. "Cooperatives, contract farming, and farm size: The case of tomato producers in Nepal," Agribusiness, John Wiley & Sons, Ltd., vol. 34(4), pages 865-886, October.
    13. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    14. Dehejia Rajeev, 2015. "Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 47-69, June.
    15. Marco Caliendo, 2009. "Start-up subsidies in East Germany: finally, a policy that works?," International Journal of Manpower, Emerald Group Publishing, vol. 30(7), pages 625-647, November.
    16. Czarnitzki, Dirk & Lopes-Bento, Cindy, 2013. "Value for money? New microeconometric evidence on public R&D grants in Flanders," Research Policy, Elsevier, vol. 42(1), pages 76-89.
    17. Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2017. "The finite sample performance of semi- and non-parametric estimators for treatment effects and policy evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 91-102.
    18. Eliasson, Kent, 2006. "How Robust is the Evidence on the Returns to College Choice? Results Using Swedish Administrative Data," Umeå Economic Studies 692, Umeå University, Department of Economics.
    19. Hottenrott, Hanna & Lopes-Bento, Cindy, 2014. "(International) R&D collaboration and SMEs: The effectiveness of targeted public R&D support schemes," Research Policy, Elsevier, vol. 43(6), pages 1055-1066.
    20. Marco Caliendo & Steffen Künn, 2015. "Getting back into the labor market: the effects of start-up subsidies for unemployed females," Journal of Population Economics, Springer;European Society for Population Economics, vol. 28(4), pages 1005-1043, October.

    More about this item

    Keywords

    Wage Indicator; online surveys; propensity score matching; weights;
    All these keywords.

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J30 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - General
    • J31 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - Wage Level and Structure; Wage Differentials

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fme:wpaper:26. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/grauwpl.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jan Hagemejer (email available below). General contact details of provider: https://edirc.repec.org/data/grauwpl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.