IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11424.html
   My bibliography  Save this paper

Skating on Thin Evidence: Implications for Public Policy

Author

Listed:
  • Doucouliagos, Chris

    () (Deakin University)

  • Paldam, Martin

    () (Aarhus University)

  • Stanley, T. D.

    () (Deakin University)

Abstract

Good public policy needs to be evidence based. However, the evidence base is thin for many policy issues. How can policy makers best respond to such thin areas of research that are also quite likely to change over time? Our survey investigates the evolution of the econometric evidence base for 101 economic issues, drawing upon 42,578 effect sizes (mainly elasticities and correlations) from 4,300 econometric studies. We evaluate the performance of six approaches to early research assessment: the simple unweighted mean; the median; the Paldam, "divide by 2" rule of thumb; the unrestricted weighted least squares (WLS) weighted average; the PET-PEESE meta-regression correction for publication bias; the weighted average of the adequately powered (WAAP); and WAAP-WLS. Lowest prediction errors are found in the Paldam rule of thumb and WLS. WLS typically reduces the initial exaggeration of thin evidence by half.

Suggested Citation

  • Doucouliagos, Chris & Paldam, Martin & Stanley, T. D., 2018. "Skating on Thin Evidence: Implications for Public Policy," IZA Discussion Papers 11424, Institute of Labor Economics (IZA).
  • Handle: RePEc:iza:izadps:dp11424
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp11424.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Heckman, James J., 2000. "Microdata, Heterogeneity and the Evaluation of Public Policy," Nobel Prize in Economics documents 2000-4, Nobel Prize Committee.
    2. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    3. Viscusi, W Kip & Aldy, Joseph E, 2003. "The Value of a Statistical Life: A Critical Review of Market Estimates throughout the World," Journal of Risk and Uncertainty, Springer, vol. 27(1), pages 5-76, August.
    4. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    5. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    6. T.D. Stanley, 2013. "Does economics add up? An introduction to meta-regression analysis," European Journal of Economics and Economic Policies: Intervention, Edward Elgar Publishing, vol. 10(2), pages 207-220.
    7. Evanschitzky, Heiner & Armstrong, J. Scott, 2010. "Replications of forecasting research," International Journal of Forecasting, Elsevier, vol. 26(1), pages 4-8, January.
    8. T. D. Stanley, 2005. "Integrating the Empirical Tests of the Natural Rate Hypothesis: A Meta‐Regression Analysis," Kyklos, Wiley Blackwell, vol. 58(4), pages 611-634, November.
    9. repec:aea:jeclit:v:56:y:2018:i:3:p:920-80 is not listed on IDEAS
    10. Tomáš Havránek, 2010. "Rose effect and the euro: is the magic gone?," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 146(2), pages 241-261, June.
    11. repec:reg:rpubli:282 is not listed on IDEAS
    12. Charles N. Bailey & James R. Hasselback & Julia N. Karcher, 2001. "Research Misconduct in Accounting Literature: A Survey of the Most Prolific Researchers’ Actions and Beliefs," Abacus, Accounting Foundation, University of Sydney, vol. 37(1), pages 26-54, February.
    13. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    14. Raymond Hubbard & Daniel E. Vetter, 1992. "The Publication Incidence of Replications and Critical Commentary in Economics," The American Economist, Sage Publications, vol. 36(1), pages 29-34, March.
    15. repec:wly:econjl:v:127:y:2017:i:605:p:f236-f265 is not listed on IDEAS
    16. T. D. Stanley, 2004. "Does unemployment hysteresis falsify the natural rate hypothesis? a meta-regression analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 18(4), pages 589-612, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Martin Paldam, 2019. "Methods used in economic research. An empirical study of trends and levels," Working Papers CEB 19-002, ULB -- Universite Libre de Bruxelles.
    2. repec:eee:poleco:v:54:y:2018:i:c:p:5-15 is not listed on IDEAS

    More about this item

    Keywords

    WLS; meta-regression; thin evidence; WAAP; Paldam rule of thumb;

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • H00 - Public Economics - - General - - - General
    • H5 - Public Economics - - National Government Expenditures and Related Policies

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11424. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Holger Hinte). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.