IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/17-04.html
   My bibliography  Save this paper

Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?

Author

Listed:

Abstract

This paper studies the performance of the FAT-PET-PEESE (FPP) procedure, a commonly employed approach for addressing publication bias in the economics and business meta-analysis literature. The FPP procedure is generally used for three purposes: (i) to test whether a sample of estimates suffers from publication bias, (ii) to test whether the estimates indicate that the effect of interest is statistically different from zero, and (iii) to obtain an estimate of the mean true effect. Our findings indicate that the FPP procedure performs well in the basic but unrealistic environment of “fixed effects,” where all estimates are assumed to derive from a single population value and sampling error is the only reason for why studies produce different estimates. However, when we study its performance in more realistic data environments, where there is heterogeneity in the population effects across and within studies, the FPP procedure becomes unreliable for the first two purposes, and is less efficient than other estimators when estimating overall mean effect. Further, hypothesis tests about the mean true effect are frequently unreliable. We corroborate our findings by recreating the simulation framework of Stanley and Doucouliagos (2017) and repeat our tests using their framework.

Suggested Citation

  • Nazila Alinaghi & W. Robert Reed, 2017. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 17/04, University of Canterbury, Department of Economics and Finance.
  • Handle: RePEc:cbt:econwp:17/04
    as

    Download full text from publisher

    File URL: http://www.econ.canterbury.ac.nz/RePEc/cbt/econwp/1704.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Gunby, Philip & Jin, Yinghua & Robert Reed, W., 2017. "Did FDI Really Cause Chinese Economic Growth? A Meta-Analysis," World Development, Elsevier, vol. 90(C), pages 242-255.
    2. Carla Haelermans & Lex Borghans, 2012. "Wage Effects of On-the-Job Training: A Meta-Analysis," British Journal of Industrial Relations, London School of Economics, vol. 50(3), pages 502-528, September.
    3. Mora Rodríguez, Jhon James & Muro, Juan, 2015. "On the size of sheepskin effects: A meta-analysis," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 9, pages 1-18.
    4. T. D. Stanley, 2008. "Meta-Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    5. Hristos Doucouliagos & Janto Haman & T.D. Stanley, 2012. "Pay for Performance and Corporate Governance Reform," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 51(3), pages 670-703, July.
    6. Reed, W. Robert, 2015. "A Monte Carlo analysis of alternative meta-analysis estimators in the presence of publication bias," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 9, pages 1-40.
    7. repec:bla:brjirl:v:54:y:2016:i:4:p:709-741 is not listed on IDEAS
    8. Lazzaroni, Sara & van Bergeijk, Peter A.G., 2014. "Natural disasters' impact, factors of resilience and development: A meta-analysis of the macroeconomic literature," Ecological Economics, Elsevier, vol. 107(C), pages 333-346.
    9. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum-Wage Research? A Meta-Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    10. Efendic, Adnan & Pugh, Geoff & Adnett, Nick, 2011. "Institutions and economic performance: A meta-regression analysis," European Journal of Political Economy, Elsevier, vol. 27(3), pages 586-599, September.
    11. Joan Costa‐Font & Marin Gemmill & Gloria Rubert, 2011. "Biases in the healthcare luxury good hypothesis?: a meta‐regression analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(1), pages 95-107, January.
    12. Jon Nelson, 2013. "Meta-analysis of alcohol price and income elasticities – with corrections for publication bias," Health Economics Review, Springer, vol. 3(1), pages 1-10, December.
    13. Megan Linde Leonard & T. D. Stanley & Hristos Doucouliagos, 2014. "Does the UK Minimum Wage Reduce Employment? A Meta-Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 52(3), pages 499-520, September.
    14. Doucouliagos, Chris & Stanley, T.D. & Giles, Margaret, 2012. "Are estimates of the value of a statistical life exaggerated?," Journal of Health Economics, Elsevier, vol. 31(1), pages 197-206.
    15. Tomáš Havránek, 2010. "Rose effect and the euro: is the magic gone?," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 146(2), pages 241-261, June.
    16. Iwasaki, Ichiro & Tokunaga, Masahiro, 2014. "Macroeconomic Impacts of FDI in Transition Economies: A Meta-Analysis," World Development, Elsevier, vol. 61(C), pages 53-69.
    17. Hristos Doucouliagos & Martin Paldam, 2013. "The Robust Result in Meta-analysis of Aid Effectiveness: A Response to Mekasha and Tarp," Journal of Development Studies, Taylor & Francis Journals, vol. 49(4), pages 584-587, April.
    18. Chliova, Myrto & Brinckmann, Jan & Rosenbusch, Nina, 2015. "Is microcredit a blessing for the poor? A meta-analysis examining development outcomes and contextual considerations," Journal of Business Venturing, Elsevier, vol. 30(3), pages 467-487.
    19. Doucouliagos, Hristos & Stanley, T.D. & Viscusi, W. Kip, 2014. "Publication selection and the income elasticity of the value of a statistical life," Journal of Health Economics, Elsevier, vol. 33(C), pages 67-75.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    Meta-analysis; publication bias; funnel asymmetry test (FAT); Precision Effect Estimate with Standard Error (PEESE); Monte Carlo; Simulations;

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:17/04. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Albert Yee). General contact details of provider: http://edirc.repec.org/data/decannz.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.