IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11424.html
   My bibliography  Save this paper

Skating on Thin Evidence: Implications for Public Policy

Author

Listed:
  • Doucouliagos, Chris

    () (Deakin University)

  • Paldam, Martin

    () (Aarhus University)

  • Stanley, T. D.

    () (Deakin University)

Abstract

Good public policy needs to be evidence based. However, the evidence base is thin for many policy issues. How can policy makers best respond to such thin areas of research that are also quite likely to change over time? Our survey investigates the evolution of the econometric evidence base for 101 economic issues, drawing upon 42,578 effect sizes (mainly elasticities and correlations) from 4,300 econometric studies. We evaluate the performance of six approaches to early research assessment: the simple unweighted mean; the median; the Paldam, "divide by 2" rule of thumb; the unrestricted weighted least squares (WLS) weighted average; the PET-PEESE meta-regression correction for publication bias; the weighted average of the adequately powered (WAAP); and WAAP-WLS. Lowest prediction errors are found in the Paldam rule of thumb and WLS. WLS typically reduces the initial exaggeration of thin evidence by half.

Suggested Citation

  • Doucouliagos, Chris & Paldam, Martin & Stanley, T. D., 2018. "Skating on Thin Evidence: Implications for Public Policy," IZA Discussion Papers 11424, Institute for the Study of Labor (IZA).
  • Handle: RePEc:iza:izadps:dp11424
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp11424.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    2. Viscusi, W Kip & Aldy, Joseph E, 2003. "The Value of a Statistical Life: A Critical Review of Market Estimates throughout the World," Journal of Risk and Uncertainty, Springer, vol. 27(1), pages 5-76, August.
    3. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    4. T.D. Stanley, 2013. "Does economics add up? An introduction to meta-regression analysis," European Journal of Economics and Economic Policies: Intervention, Edward Elgar Publishing, vol. 10(2), pages 207-220.
    5. Evanschitzky, Heiner & Armstrong, J. Scott, 2010. "Replications of forecasting research," International Journal of Forecasting, Elsevier, vol. 26(1), pages 4-8, January.
    6. T. D. Stanley, 2005. "Integrating the Empirical Tests of the Natural Rate Hypothesis: A Meta-Regression Analysis," Kyklos, Wiley Blackwell, vol. 58(4), pages 611-634, November.
    7. repec:reg:rpubli:282 is not listed on IDEAS
    8. Charles N. Bailey & James R. Hasselback & Julia N. Karcher, 2001. "Research Misconduct in Accounting Literature: A Survey of the Most Prolific Researchers' Actions and Beliefs," Abacus, Accounting Foundation, University of Sydney, vol. 37(1), pages 26-54.
    9. Garret S. Christensen & Edward Miguel, 2016. "Transparency, Reproducibility, and the Credibility of Economics Research," NBER Working Papers 22989, National Bureau of Economic Research, Inc.
    10. Raymond Hubbard & Daniel E. Vetter, 1992. "The Publication Incidence of Replications and Critical Commentary in Economics," The American Economist, Sage Publications, vol. 36(1), pages 29-34, March.
    11. T. D. Stanley, 2004. "Does unemployment hysteresis falsify the natural rate hypothesis? a meta-regression analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 18(4), pages 589-612, September.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    WLS; meta-regression; thin evidence; WAAP; Paldam rule of thumb;

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • H00 - Public Economics - - General - - - General
    • H5 - Public Economics - - National Government Expenditures and Related Policies

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11424. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.