The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment
AbstractIn this paper we use Monte Carlo simulation to investigate the impact of effect size heterogeneity on the results of a meta-analysis. Specifically, we address the small sample behaviour of the OLS, the fixed effects regression and the mixed effects meta-estimators under three alternative scenarios of effect size heterogeneity. We distinguish heterogeneity in effect size variance, heterogeneity due to a varying true underlying effect across primary studies, and heterogeneity due to a non-systematic impact of omitted variable bias in primary studies. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations. However, in the presence of random effect size variation due to a non-systematic impact of omitted variable bias, using the mixed effects estimator may be suboptimal. We also address the impact of sample size and show that meta-analysis sample size is far more effective in reducing meta-estimator variance and increasing the power of hypothesis testing than primary study sample size.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Tinbergen Institute in its series Tinbergen Institute Discussion Papers with number 07-052/3.
Date of creation: 16 Jul 2007
Date of revision:
Contact details of provider:
Web page: http://www.tinbergen.nl
Effect size heterogeneity; meta-analysis; Monte Carlo simulation; fixed effects regression estimator; mixed effects estimator;
Find related papers by JEL classification:
- C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
- C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
- C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General
This paper has been announced in the following NEP Reports:
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Robert Goldfarb, 1995. "The economist-as-audience needs a methodology of plausible inference," Journal of Economic Methodology, Taylor & Francis Journals, vol. 2(2), pages 201-222.
- Smith, V Kerry & Huang, Ju-Chin, 1995. "Can Markets Value Air Quality? A Meta-analysis of Hedonic Property Value Models," Journal of Political Economy, University of Chicago Press, vol. 103(1), pages 209-27, February.
- Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-43, May.
- Mark J. Koetse & Raymond J.G.M. Florax & Henri L.F. de Groot, 2005. "Correcting for Primary Study Misspecifications in Meta-Analysis," Tinbergen Institute Discussion Papers 05-029/3, Tinbergen Institute, revised 31 Jan 2013.
- Julio Sánchez-Meca & Fulgencio Marín-Martínez, 1997. "Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error," Quality & Quantity: International Journal of Methodology, Springer, vol. 31(4), pages 385-399, November.
- Ian J. Bateman & Andrew P. Jones, 2003. "Contrasting Conventional with Multi-Level Modeling Approaches to Meta-Analysis: Expectation Consistency in U.K. Woodland Recreation Values," Land Economics, University of Wisconsin Press, vol. 79(2), pages 235-258.
- T. D. Stanley, 2001. "Wheat from Chaff: Meta-analysis as Quantitative Literature Review," Journal of Economic Perspectives, American Economic Association, vol. 15(3), pages 131-150, Summer.
- Gorg, Holger & Strobl, Eric, 2001. "Multinational Companies and Productivity Spillovers: A Meta-analysis," Economic Journal, Royal Economic Society, vol. 111(475), pages F723-39, November.
- Jon Nelson & Peter Kennedy, 2009. "The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment," Environmental & Resource Economics, European Association of Environmental and Resource Economists, vol. 42(3), pages 345-377, March.
- Ceren Ozgen & Peter Nijkamp & Jacques Poot, 2010.
"The effect of migration on income growth and convergence: Meta-analytic evidence,"
Papers in Regional Science,
Wiley Blackwell, vol. 89(3), pages 537-561, 08.
- Ozgen, Ceren & Nijkamp, Peter & Poot, Jacques, 2009. "The Effect of Migration on Income Growth and Convergence: Meta-Analytic Evidence," IZA Discussion Papers 4522, Institute for the Study of Labor (IZA).
- Tomáš Havránek, 2009. "Rose Effect and the Euro: The Magic is Gone," Working Papers IES 2009/20, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Aug 2009.
- Melo, Patricia C. & Graham, Daniel J. & Noland, Robert B., 2009. "A meta-analysis of estimates of urban agglomeration economies," Regional Science and Urban Economics, Elsevier, vol. 39(3), pages 332-342, May.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Antoine Maartens (+31 626 - 160 892)).
If references are entirely missing, you can add them using this form.