IDEAS home Printed from https://ideas.repec.org/a/spr/aistmt/v72y2020i2d10.1007_s10463-018-0691-8.html
   My bibliography  Save this article

Strong model dependence in statistical analysis: goodness of fit is not enough for model choice

Author

Listed:
  • John Copas

    (University of Warwick)

  • Shinto Eguchi

    (Institute of Statistical Mathematics)

Abstract

Most statistical methods are based on models, but most practical applications ignore the fact that the results depend on the model as well as on the data. This paper examines the size of this model dependence, and finds that there can be very considerable variation between the results of fitting different models to the same data, even if the models being considered are restricted to those which give an acceptable fit to the data. Under reasonable regularity conditions, we show that different empirically acceptable models can give rise to non-overlapping confidence intervals for the same parameter. Application papers need to recognize that the validity of conventional statistical results rests on the assumption that the underlying model is known to be correct, and that this is a much stronger requirement than merely confirming that the model gives a good fit to the data. The problem of model dependence is only partially resolved by using formal methods of model selection or model averaging.

Suggested Citation

  • John Copas & Shinto Eguchi, 2020. "Strong model dependence in statistical analysis: goodness of fit is not enough for model choice," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(2), pages 329-352, April.
    • Handle: RePEc:spr:aistmt:v:72:y:2020:i:2:d:10.1007_s10463-018-0691-8
    DOI: 10.1007/s10463-018-0691-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10463-018-0691-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10463-018-0691-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Bradley Efron, 2014. "Estimation and Accuracy After Model Selection," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 991-1007, September.
    3. Leeb, Hannes & Pötscher, Benedikt M., 2005. "Model Selection And Inference: Facts And Fiction," Econometric Theory, Cambridge University Press, vol. 21(1), pages 21-59, February.
    4. Hjort N.L. & Claeskens G., 2003. "Frequentist Model Average Estimators," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 879-899, January.
    5. Claeskens,Gerda & Hjort,Nils Lid, 2008. "Model Selection and Model Averaging," Cambridge Books, Cambridge University Press, number 9780521852258.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giuseppe De Luca & Jan Magnus & Franco Peracchi, 2022. "Asymptotic properties of the weighted average least squares (WALS) estimator," Tinbergen Institute Discussion Papers 22-022/III, Tinbergen Institute.
    2. Lenard Lieb & Stephan Smeekes, 2017. "Inference for Impulse Responses under Model Uncertainty," Papers 1709.09583, arXiv.org, revised Oct 2019.
    3. DiTraglia, Francis J., 2016. "Using invalid instruments on purpose: Focused moment selection and averaging for GMM," Journal of Econometrics, Elsevier, vol. 195(2), pages 187-208.
    4. Magnus, Jan R. & Wan, Alan T.K. & Zhang, Xinyu, 2011. "Weighted average least squares estimation with nonspherical disturbances and an application to the Hong Kong housing market," Computational Statistics & Data Analysis, Elsevier, vol. 55(3), pages 1331-1341, March.
    5. Ruoyao Shi, 2021. "An Averaging Estimator for Two Step M Estimation in Semiparametric Models," Working Papers 202105, University of California at Riverside, Department of Economics.
    6. Liu, Chu-An & Tao, Jing, 2016. "Model selection and model averaging in nonparametric instrumental variables models," MPRA Paper 69492, University Library of Munich, Germany.
    7. Ali Charkhi & Gerda Claeskens, 2018. "Asymptotic post-selection inference for the Akaike information criterion," Biometrika, Biometrika Trust, vol. 105(3), pages 645-664.
    8. Wan, Alan T.K. & Zhang, Xinyu & Zou, Guohua, 2010. "Least squares model averaging by Mallows criterion," Journal of Econometrics, Elsevier, vol. 156(2), pages 277-283, June.
    9. Uwe Hassler & Marc‐Oliver Pohle, 2022. "Unlucky Number 13? Manipulating Evidence Subject to Snooping," International Statistical Review, International Statistical Institute, vol. 90(2), pages 397-410, August.
    10. Tumala, Mohammed M & Olubusoye, Olusanya E & Yaaba, Baba N & Yaya, OlaOluwa S & Akanbi, Olawale B, 2017. "Forecasting Nigerian Inflation using Model Averaging methods: Modelling Frameworks to Central Banks," MPRA Paper 88754, University Library of Munich, Germany, revised Feb 2018.
    11. Shaobo Jin, 2022. "Frequentist Model Averaging in Structure Equation Model With Ordinal Data," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1130-1145, September.
    12. Jing Zhou & Gerda Claeskens & Jelena Bradic, 2020. "Detangling robustness in high dimensions: composite versus model-averaged estimation," Papers 2006.07457, arXiv.org.
    13. Leeb, Hannes & Pötscher, Benedikt M., 2008. "Can One Estimate The Unconditional Distribution Of Post-Model-Selection Estimators?," Econometric Theory, Cambridge University Press, vol. 24(2), pages 338-376, April.
    14. Phillip Heiler & Jana Mareckova, 2019. "Shrinkage for Categorical Regressors," Papers 1901.01898, arXiv.org.
    15. Fletcher, David & Dillingham, Peter W., 2011. "Model-averaged confidence intervals for factorial experiments," Computational Statistics & Data Analysis, Elsevier, vol. 55(11), pages 3041-3048, November.
    16. Yuting Wei & Qihua Wang & Wei Liu, 2021. "Model averaging for linear models with responses missing at random," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 73(3), pages 535-553, June.
    17. Katrin Wölfel & Christoph S. Weber, 2017. "Searching for the Fed’s reaction function," Empirical Economics, Springer, vol. 52(1), pages 191-227, February.
    18. Christian T. Brownlees & Giampiero Gallo, 2007. "Volatility Forecasting Using Explanatory Variables and Focused Selection Criteria," Econometrics Working Papers Archive wp2007_04, Universita' degli Studi di Firenze, Dipartimento di Statistica, Informatica, Applicazioni "G. Parenti".
    19. Su, Jiun-Hua, 2021. "Model selection in utility-maximizing binary prediction," Journal of Econometrics, Elsevier, vol. 223(1), pages 96-124.
    20. Spanos, Aris, 2010. "Akaike-type criteria and the reliability of inference: Model selection versus statistical model specification," Journal of Econometrics, Elsevier, vol. 158(2), pages 204-220, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:aistmt:v:72:y:2020:i:2:d:10.1007_s10463-018-0691-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.