IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v46y2017i1p3-40.html
   My bibliography  Save this article

Model Uncertainty and Robustness

Author

Listed:
  • Cristobal Young
  • Katherine Holsteen

Abstract

Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.

Suggested Citation

  • Cristobal Young & Katherine Holsteen, 2017. "Model Uncertainty and Robustness," Sociological Methods & Research, , vol. 46(1), pages 3-40, January.
    • Handle: RePEc:sae:somere:v:46:y:2017:i:1:p:3-40
    DOI: 10.1177/0049124115610347
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124115610347
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124115610347?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    2. Antonio Ciccone & Marek Jarociński, 2010. "Determinants of Economic Growth: Will Data Tell?," American Economic Journal: Macroeconomics, American Economic Association, vol. 2(4), pages 222-246, October.
    3. Andrew Leigh, 2007. "How Closely Do Top Income Shares Track Other Measures of Inequality?," Economic Journal, Royal Economic Society, vol. 117(524), pages 619-633, November.
    4. Henrik Jacobsen Kleven & Camille Landais & Emmanuel Saez, 2013. "Taxation and International Migration of Superstars: Evidence from the European Football Market," American Economic Review, American Economic Association, vol. 103(5), pages 1892-1924, August.
    5. Steven N. Durlauf & Chao Fu & Salvador Navarro, 2012. "Assumptions Matter: Model Uncertainty and the Deterrent Effect of Capital Punishment," American Economic Review, American Economic Association, vol. 102(3), pages 487-492, May.
    6. Jennifer Montez & Robert Hummer & Mark Hayward, 2012. "Educational Attainment and Adult Mortality in the United States: A Systematic Analysis of Functional Form," Demography, Springer;Population Association of America (PAA), vol. 49(1), pages 315-336, February.
    7. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    8. Jennifer Kane & S. Morgan & Kathleen Harris & David Guilkey, 2013. "The Educational Consequences of Teen Childbearing," Demography, Springer;Population Association of America (PAA), vol. 50(6), pages 2129-2150, December.
    9. Conway, Karen Smith & Rork, Jonathan C., 2012. "No Country for Old Men (Or Women) — Do State Tax Policies Drive Away the Elderly?," National Tax Journal, National Tax Association;National Tax Journal, vol. 65(2), pages 313-356, June.
    10. Philippe Aghion & Steven Durlauf (ed.), 2005. "Handbook of Economic Growth," Handbook of Economic Growth, Elsevier, edition 1, volume 1, number 1.
    11. Young, Cristobal & Varner, Charles, 2011. "Millionaire Migration and State Taxation of Top Incomes: Evidence From a Natural Experiment," National Tax Journal, National Tax Association;National Tax Journal, vol. 64(2), pages 255-283, June.
    12. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    13. Xavier Sala-I-Martin & Gernot Doppelhofer & Ronald I. Miller, 2004. "Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach," American Economic Review, American Economic Association, vol. 94(4), pages 813-835, September.
    14. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James R. Brown & J Anthony Cookson & Rawley Heimer, 2014. "Legal Institutions, Credit Markets, and Economic Activity," Working Papers (Old Series) 1434, Federal Reserve Bank of Cleveland.
    2. M. Shahe Emran & Fenohasina Maret-Rakotondrazaka & Stephen C. Smith, 2014. "Education and Freedom of Choice: Evidence from Arranged Marriages in Vietnam," Journal of Development Studies, Taylor & Francis Journals, vol. 50(4), pages 481-501, April.
    3. Boateng, Elliot & Agbola, Frank W. & Mahmood, Amir, 2021. "Foreign aid volatility and economic growth in Sub-Saharan Africa: Does institutional quality matter?," Economic Modelling, Elsevier, vol. 96(C), pages 111-127.
    4. Bauernschuster, Stefan & Falck, Oliver & Woessmann, Ludger, 2014. "Surfing alone? The internet and social capital: Evidence from an unforeseeable technological mistake," Journal of Public Economics, Elsevier, vol. 117(C), pages 73-89.
    5. Dimitrios Rovithis, 2013. "Do health economic evaluations using observational data provide reliable assessment of treatment effects?," Health Economics Review, Springer, vol. 3(1), pages 1-7, December.
    6. Mark Carlson & Kris James Mitchener, 2009. "Branch Banking as a Device for Discipline: Competition and Bank Survivorship during the Great Depression," Journal of Political Economy, University of Chicago Press, vol. 117(2), pages 165-210, April.
    7. Ilona Babenko & Benjamin Bennett & John M Bizjak & Jeffrey L Coles & Jason J Sandvik, 2023. "Clawback Provisions and Firm Risk," The Review of Corporate Finance Studies, Society for Financial Studies, vol. 12(2), pages 191-239.
    8. Ma, Lingjie & Koenker, Roger, 2006. "Quantile regression methods for recursive structural equation models," Journal of Econometrics, Elsevier, vol. 134(2), pages 471-506, October.
    9. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    10. Messer, Dolores & Wolter, Stefan C., 2005. "Are Student Exchange Programs Worth It?," IZA Discussion Papers 1656, Institute of Labor Economics (IZA).
    11. Aaron Jackson & William Miles, 2008. "Fixed Exchange Rates and Disinflation in Emerging Markets: How Large Is the Effect?," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 144(3), pages 538-557, October.
    12. Berthélemy Michel & Bonev Petyo & Dussaux Damien & Söderberg Magnus, 2019. "Methods for strengthening a weak instrument in the case of a persistent treatment," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 23(1), pages 1-30, February.
    13. Fali Huang & Myoung-Jae Lee, 2010. "Dynamic treatment effect analysis of TV effects on child cognitive development," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(3), pages 392-419.
    14. Vitalis, Kyriacos & Stefanidis, Dimosthenis & Pallis, George & Dikaiakos, Marios & Nicolaou, Nicos & Nicolaides, Christos, 2024. "Quantifying the impact of online social networks on the success of entrepreneurs," OSF Preprints x6vda_v1, Center for Open Science.
    15. Richard Harris & John Moffat, 2011. "R&D, Innovation and Exporting," SERC Discussion Papers 0073, Centre for Economic Performance, LSE.
    16. Hope Corman & Dhaval Dave & Nancy E. Reichman, 2018. "Evolution of the Infant Health Production Function," Southern Economic Journal, John Wiley & Sons, vol. 85(1), pages 6-47, July.
    17. Steven A. Boutcher & Jason N. Houle & Anna Raup‐Kounovksy & Carroll Seron, 2023. "A Faustian bargain? Rethinking the role of debt in law students' career choices," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 20(1), pages 166-195, March.
    18. Wu, Fang & Cao, June & Zhang, Xiaosan, 2023. "Do non-executive employees matter in curbing corporate financial fraud?," Journal of Business Research, Elsevier, vol. 163(C).
    19. Katherine Sawyer & Kathleen Gallagher Cunningham & William Reed, 2017. "The Role of External Support in Civil War Termination," Journal of Conflict Resolution, Peace Science Society (International), vol. 61(6), pages 1174-1202, July.
    20. Yiming He & Thomas M. Fullerton, 2020. "The economic analysis of instrument variables estimation in dynamic optimal models with an application to the water consumption," Agricultural Economics, Czech Academy of Agricultural Sciences, vol. 66(9), pages 413-423.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:46:y:2017:i:1:p:3-40. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.