IDEAS home Printed from https://ideas.repec.org/p/tin/wpaper/20050029.html
   My bibliography  Save this paper

Correcting for Primary Study Misspecifications in Meta-Analysis

Author

Listed:
  • Mark J. Koetse

    (Faculty of Economics and Business Administration, Vrije Universiteit Amsterdam)

  • Raymond J.G.M. Florax

    (Vrije Universiteit, and Purdue University)

  • Henri L.F. de Groot

    (Faculty of Economics and Business Administration, Vrije Universiteit Amsterdam)

Abstract

In non-experimental sciences the errors associated with model misspecifications in primarystudies carry over to meta-analysis. We use Monte Carlo simulations to analyse the effects ofthese misspecifications on results of a meta-analysis using a meta-estimator that calculates asimple average effect and a meta-estimator that includes dummy variables to control forprimary study misspecification. The results show that using the dummy variable model goes along way in mitigating the negative effects of error propagation on the bias and mean squarederror of the meta-estimator, and the size and the power of statistical tests. Although primarystudy misspecifications can usually be observed and controlled for in a meta-analysis, themore complex interactions between these observed characteristics are far more difficult tocontrol for in practice. Our results show that these interactions may, however, substantiallyaffect the outcomes of a meta-analysis. When meta-analysis is used to look for a ‘true’ effectrather than for analysing variation in outcomes, our results provide an argument for studyselection on model quality to avoid the impact of error propagation in meta-analysis.

Suggested Citation

  • Mark J. Koetse & Raymond J.G.M. Florax & Henri L.F. de Groot, 2005. "Correcting for Primary Study Misspecifications in Meta-Analysis," Tinbergen Institute Discussion Papers 05-029/3, Tinbergen Institute, revised 31 Jan 2013.
    • Handle: RePEc:tin:wpaper:20050029
    as

    Download full text from publisher

    File URL: https://papers.tinbergen.nl/05029.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mark Koetse & Raymond Florax & Henri Groot, 2010. "Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 19(2), pages 217-236, June.
    2. Jasper M. Dalhuisen & Raymond J. G. M. Florax & JHenri L. F. de Groot & Peter Nijkamp, 2003. "Price and Income Elasticities of Residential Water Demand: A Meta-Analysis," Land Economics, University of Wisconsin Press, vol. 79(2), pages 292-308.
    3. Kluve, Jochen, 2010. "The effectiveness of European active labor market programs," Labour Economics, Elsevier, vol. 17(6), pages 904-918, December.
    4. Lars P. Feld & Jost H. Heckemeyer, 2011. "Fdi And Taxation: A Meta‐Study," Journal of Economic Surveys, Wiley Blackwell, vol. 25(2), pages 233-272, April.
    5. Anne-Célia Disdier & Keith Head, 2008. "The Puzzling Persistence of the Distance Effect on Bilateral Trade," The Review of Economics and Statistics, MIT Press, vol. 90(1), pages 37-48, February.
    6. Julio Sánchez-Meca & Fulgencio Marín-Martínez, 1997. "Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error," Quality & Quantity: International Journal of Methodology, Springer, vol. 31(4), pages 385-399, November.
    7. Hristos Doucouliagos & Martin Paldam, 2009. "The Aid Effectiveness Literature: The Sad Results Of 40 Years Of Research," Journal of Economic Surveys, Wiley Blackwell, vol. 23(3), pages 433-461, July.
    8. Viscusi, W Kip & Aldy, Joseph E, 2003. "The Value of a Statistical Life: A Critical Review of Market Estimates throughout the World," Journal of Risk and Uncertainty, Springer, vol. 27(1), pages 5-76, August.
    9. Raymond J.G.M. Florax & Henri L.F. de Groot & Peter Mulder (ed.), 2011. "Improving Energy Efficiency through Technology," Books, Edward Elgar Publishing, number 3830.
    10. Jon Nelson & Peter Kennedy, 2009. "The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 42(3), pages 345-377, March.
    11. Ai, Chunrong & Norton, Edward C., 2003. "Interaction terms in logit and probit models," Economics Letters, Elsevier, vol. 80(1), pages 123-129, July.
    12. repec:reg:rpubli:282 is not listed on IDEAS
    13. Rick L. Williams, 2000. "A Note on Robust Variance Estimation for Cluster-Correlated Data," Biometrics, The International Biometric Society, vol. 56(2), pages 645-646, June.
    14. Peter Kennedy, 2003. "A Guide to Econometrics, 5th Edition," MIT Press Books, The MIT Press, edition 5, volume 1, number 026261183x, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. G. Cornelis van Kooten & Susanna Laaksonen-Craig & Yichuan Wang, 2007. "Costs of Creating Carbon Offset Credits via Forestry Activities: A Meta-Regression Analysis," Working Papers 2007-03, University of Victoria, Department of Economics, Resource Economics and Policy Analysis Research Group.
    2. Entorf, Horst & Hou, Jia, 2018. "Financial Education for the Disadvantaged? A Review," IZA Discussion Papers 11515, Institute of Labor Economics (IZA).
    3. Randall S. Rosenberger & Robert J. Johnston, 2009. "Selection Effects in Meta-Analysis and Benefit Transfer: Avoiding Unintended Consequences," Land Economics, University of Wisconsin Press, vol. 85(3), pages 410-428.
    4. Jon Nelson & Peter Kennedy, 2009. "The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 42(3), pages 345-377, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:wly:econjl:v::y:2017:i:605:p:f236-f265 is not listed on IDEAS
    2. Jon Nelson & Peter Kennedy, 2009. "The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 42(3), pages 345-377, March.
    3. Francesco Aiello & Graziella Bonanno, 2016. "Efficiency in banking: a meta-regression analysis," International Review of Applied Economics, Taylor & Francis Journals, vol. 30(1), pages 112-149, January.
    4. Reed, W. Robert, 2015. "A Monte Carlo analysis of alternative meta-analysis estimators in the presence of publication bias," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-40.
    5. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    6. Liu, Boying & Richard Shumway, C., 2016. "Substitution elasticities between GHG-polluting and nonpolluting inputs in agricultural production: A meta-regression," Energy Economics, Elsevier, vol. 54(C), pages 123-132.
    7. Binyam A. Demena & Peter A. G. van Bergeijk, 2017. "A Meta-Analysis Of Fdi And Productivity Spillovers In Developing Countries," Journal of Economic Surveys, Wiley Blackwell, vol. 31(2), pages 546-571, April.
    8. Francesco Aiello & Graziella Bonanno & Francesco Foglia, 2021. "On The Heterogeneity In The Judicial Efficiency Literature: A Meta-Regression Analysis," Working Papers 202102, Università della Calabria, Dipartimento di Economia, Statistica e Finanza "Giovanni Anania" - DESF.
    9. Francesco Aiello & Graziella Bonanno & Luigi Capristo, 2017. "Explaining Differences In Efficiency: The Case Of Local Government Literature," Working Papers 201704, Università della Calabria, Dipartimento di Economia, Statistica e Finanza "Giovanni Anania" - DESF.
    10. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    11. Dreher, Axel & Nunnenkamp, Peter & Thiel, Susann & Thiele, Rainer, 2010. "Aid allocation by German NGOs: Does the degree of public refinancing matter?," University of Göttingen Working Papers in Economics 92, University of Goettingen, Department of Economics.
    12. Stanley, T.D. & Doucouliagos, Chris & Jarrell, Stephen B., 2008. "Meta-regression analysis as the socio-economics of economics research," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 37(1), pages 276-292, February.
    13. Francisco Guijarro, 2018. "Economic Recovery and Effectiveness of Active Labour Market Initiatives for the Unemployed in Spain: A Gender Perspective of the Valencian Region," Sustainability, MDPI, vol. 10(10), pages 1-15, October.
    14. Åsa Johansson, 2016. "Public Finance, Economic Growth and Inequality: A Survey of the Evidence," OECD Economics Department Working Papers 1346, OECD Publishing.
    15. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta-Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.
    16. Dimos, Christos & Pugh, Geoff & Hisarciklilar, Mehtap & Talam, Ema & Jackson, Ian, 2022. "The relative effectiveness of R&D tax credits and R&D subsidies: A comparative meta-regression analysis," Technovation, Elsevier, vol. 115(C).
    17. Lohwasser, Todor S., 2020. "Meta-analyzing the relative performance of venture capital-backed firms," Discussion Papers of the Institute for Organisational Economics 4/2020, University of Münster, Institute for Organisational Economics.
    18. Angelika J. Budjan & Andreas Fuchs, 2021. "Democracy and Aid Donorship," American Economic Journal: Economic Policy, American Economic Association, vol. 13(4), pages 217-238, November.
    19. Yannick BINEAU, 2010. "A Empirical Assessment of the Feldstein and Horioka Literature," EcoMod2010 259600030, EcoMod.
    20. Natina Yaduma & Mika Kortelainen & Ada Wossink, 2013. "Estimating Mortality and Economic Costs of Particulate Air Pollution in Developing Countries: The Case of Nigeria," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 54(3), pages 361-387, March.
    21. Sebri, Maamar, 2015. "Use renewables to be cleaner: Meta-analysis of the renewable energy consumption–economic growth nexus," Renewable and Sustainable Energy Reviews, Elsevier, vol. 42(C), pages 657-665.

    More about this item

    Keywords

    Meta-analysis; Monte Carlo simulation; Omitted variable bias; Elasticities; Model Misspecification;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tin:wpaper:20050029. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Tinbergen Office +31 (0)10-4088900 (email available below). General contact details of provider: https://edirc.repec.org/data/tinbenl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.