IDEAS home Printed from https://ideas.repec.org/p/tin/wpaper/20070052.html

The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment

Author

Listed:
  • Mark J. Koetse

    (Vrije Universiteit Amsterdam)

  • Raymond J.G.M. Florax

    (Purdue University, and Vrije Universiteit Amsterdam)

  • Henri L.F. de Groot

    (Vrije Universiteit Amsterdam)

Abstract

In this paper we use Monte Carlo simulation to investigate the impact of effect size heterogeneity on the results of a meta-analysis. Specifically, we address the small sample behaviour of the OLS, the fixed effects regression and the mixed effects meta-estimators under three alternative scenarios of effect size heterogeneity. We distinguish heterogeneity in effect size variance, heterogeneity due to a varying true underlying effect across primary studies, and heterogeneity due to a non-systematic impact of omitted variable bias in primary studies. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations. However, in the presence of random effect size variation due to a non-systematic impact of omitted variable bias, using the mixed effects estimator may be suboptimal. We also address the impact of sample size and show that meta-analysis sample size is far more effective in reducing meta-estimator variance and increasing the power of hypothesis testing than primary study sample size.

Suggested Citation

  • Mark J. Koetse & Raymond J.G.M. Florax & Henri L.F. de Groot, 2007. "The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment," Tinbergen Institute Discussion Papers 07-052/3, Tinbergen Institute.
    • Handle: RePEc:tin:wpaper:20070052
    as

    Download full text from publisher

    File URL: https://papers.tinbergen.nl/07052.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Julio Sánchez-Meca & Fulgencio Marín-Martínez, 1997. "Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error," Quality & Quantity: International Journal of Methodology, Springer, vol. 31(4), pages 385-399, November.
    2. Robert Goldfarb, 1995. "The economist-as-audience needs a methodology of plausible inference," Journal of Economic Methodology, Taylor & Francis Journals, vol. 2(2), pages 201-222.
    3. Holger Görg & Eric Strobl, 2016. "Multinational Companies And Productivity Spillovers: A Meta-Analysis," World Scientific Book Chapters, in: MULTINATIONAL ENTERPRISES AND HOST COUNTRY DEVELOPMENT Volume 53: World Scientific Studies in International Economics, chapter 8, pages 145-161, World Scientific Publishing Co. Pte. Ltd..
    4. Smith, V Kerry & Huang, Ju-Chin, 1995. "Can Markets Value Air Quality? A Meta-analysis of Hedonic Property Value Models," Journal of Political Economy, University of Chicago Press, vol. 103(1), pages 209-227, February.
    5. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    6. Ian J. Bateman & Andrew P. Jones, 2003. "Contrasting Conventional with Multi-Level Modeling Approaches to Meta-Analysis: Expectation Consistency in U.K. Woodland Recreation Values," Land Economics, University of Wisconsin Press, vol. 79(2), pages 235-258.
    7. Doris Weichselbaumer & Rudolf Winter‐Ebmer, 2005. "A Meta‐Analysis of the International Gender Wage Gap," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 479-511, July.
    8. T. D. Stanley, 2001. "Wheat from Chaff: Meta-analysis as Quantitative Literature Review," Journal of Economic Perspectives, American Economic Association, vol. 15(3), pages 131-150, Summer.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Melo, Patricia C. & Graham, Daniel J. & Noland, Robert B., 2009. "A meta-analysis of estimates of urban agglomeration economies," Regional Science and Urban Economics, Elsevier, vol. 39(3), pages 332-342, May.
    2. Tomáš Havránek, 2009. "Rose Effect and the Euro: The Magic is Gone," Working Papers IES 2009/20, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Aug 2009.
    3. Ceren Ozgen & Peter Nijkamp & Jacques Poot, 2010. "The effect of migration on income growth and convergence: Meta‐analytic evidence," Papers in Regional Science, Wiley Blackwell, vol. 89(3), pages 537-561, August.
    4. Jon Nelson & Peter Kennedy, 2009. "The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 42(3), pages 345-377, March.
    5. Ceren Ozgen & Peter Nijkamp & Jacques Poot, 2009. "The Effect of Migration on Income Convergence: Meta-Analytic Evidence," Tinbergen Institute Discussion Papers 09-022/3, Tinbergen Institute.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mark J. Koetse & Henri L.F. de Groot & Raymond J.G.M. Florax, 2011. "A Meta-Regression Analysis of the Investment–Uncertainty Relationship," Chapters, in: Raymond J.G.M. Florax & Henri L.F. de Groot & Peter Mulder (ed.), Improving Energy Efficiency through Technology, chapter 7, Edward Elgar Publishing.
    2. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    3. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    4. Melo, Patricia C. & Graham, Daniel J. & Noland, Robert B., 2009. "A meta-analysis of estimates of urban agglomeration economies," Regional Science and Urban Economics, Elsevier, vol. 39(3), pages 332-342, May.
    5. Tomas Havranek, 2013. "Publication Bias in Measuring Intertemporal Substitution," Working Papers IES 2013/15, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Oct 2013.
    6. Feld, Lars P. & Heckemeyer, Jost H. & Overesch, Michael, 2013. "Capital structure choice and company taxation: A meta-study," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2850-2866.
    7. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    8. Gert-Jan M. Linders, 2005. "Distance Decay in International Trade Patterns - a Meta-analysis," ERSA conference papers ersa05p679, European Regional Science Association.
    9. Maria Abreu & Henri L. F. de Groot & Raymond J. G. M. Florax, 2005. "A Meta‐Analysis of β‐Convergence: the Legendary 2%," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 389-420, July.
    10. Yannick BINEAU, 2010. "A Empirical Assessment of the Feldstein and Horioka Literature," EcoMod2010 259600030, EcoMod.
    11. Masagus M. Ridhwan & Henri L.F. de Groot & Peter Nijkamp, 2010. "The Impact of Monetary Policy on Economic Activity - Evidence from a Meta-Analysis," Tinbergen Institute Discussion Papers 10-043/3, Tinbergen Institute.
    12. Mauro L. Ghinamo, 2012. "Explaining The Variation In The Empirical Estimates Of Academic Knowledge Spillovers," Journal of Regional Science, Wiley Blackwell, vol. 52(4), pages 606-634, October.
    13. Tomas Havranek & Zuzana Irsova, 2012. "Survey Article: Publication Bias in the Literature on Foreign Direct Investment Spillovers," Journal of Development Studies, Taylor & Francis Journals, vol. 48(10), pages 1375-1396, October.
    14. Stanley, T.D. & Doucouliagos, Chris & Jarrell, Stephen B., 2008. "Meta-regression analysis as the socio-economics of economics research," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 37(1), pages 276-292, February.
    15. Neves, Pedro Cunha & Afonso, Oscar & Silva, Diana & Sochirca, Elena, 2021. "The link between intellectual property rights, innovation, and growth: A meta-analysis," Economic Modelling, Elsevier, vol. 97(C), pages 196-209.
    16. Koetse, Mark J. & de Groot, Henri L.F. & Florax, Raymond J.G.M., 2008. "Capital-energy substitution and shifts in factor demand: A meta-analysis," Energy Economics, Elsevier, vol. 30(5), pages 2236-2251, September.
    17. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    18. Markus Hang & Jerome Geyer‐Klingeberg & Andreas Rathgeber & Stefan Stöckl, 2018. "Economic Development Matters: A Meta‐Regression Analysis on the Relation between Environmental Management and Financial Performance," Journal of Industrial Ecology, Yale University, vol. 22(4), pages 720-744, August.
    19. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    20. T. D. Stanley, 2004. "Does unemployment hysteresis falsify the natural rate hypothesis? a meta‐regression analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 18(4), pages 589-612, September.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tin:wpaper:20070052. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Tinbergen Office +31 (0)10-4088900 (email available below). General contact details of provider: https://edirc.repec.org/data/tinbenl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.