IDEAS home Printed from
   My bibliography  Save this paper

A multiple testing procedure for neural network model selection


  • Michele La Rocca

    () (Dept. of Economics and Statistics, University of Salerno, Italy)

  • Cira Perna

    (Dept. of Economics and Statistics, University of Salerno, Italy)


One of the most critical issues when using neural networks is how to select appropriate network architectures for the problem at hand. Practitioners usually refer to information criteria which might lead to over-parameterized models with heavy consequence on overfitting and poor ex-post forecast accuracy. Moreover, since model selection criteria depend on sample information, their actual values are subject to statistical variations. So, to compare multiple models in terms of their out of sample predictive ability, a test procedure is needed. But, in such context there is always the possibility that any satisfactory results obtained may simply be due to chance rather than any merit inherent in the model yielding to the result. The problem can be particularly serious when using neural network models which are basically atheoretical. In this paper we propose a strategy for neural network model selection which is based on a sequence of tests and, to avoid the data snooping problem, familywise error rate is controlled by a proper technique. The procedure requires the implementation of resampling techniques in order to obtain valid asymptotic critical values for the test. Some simulations results and applications to real data are discussed.

Suggested Citation

  • Michele La Rocca & Cira Perna, 2006. "A multiple testing procedure for neural network model selection," Computing in Economics and Finance 2006 497, Society for Computational Economics.
  • Handle: RePEc:sce:scecfa:497

    Download full text from publisher

    To our knowledge, this item is not available for download. To find whether it is available, there are three options:
    1. Check below whether another version of this item is available online.
    2. Check on the provider's web page whether it is in fact available.
    3. Perform a search for a similarly titled item that would be available.

    References listed on IDEAS

    1. Phillips, P.C.B., 1986. "Understanding spurious regressions in econometrics," Journal of Econometrics, Elsevier, vol. 33(3), pages 311-340, December.
    2. Phillips, P C B, 1987. "Time Series Regression with a Unit Root," Econometrica, Econometric Society, vol. 55(2), pages 277-301, March.
    3. Jushan Bai & Pierre Perron, 1998. "Estimating and Testing Linear Models with Multiple Structural Changes," Econometrica, Econometric Society, vol. 66(1), pages 47-78, January.
    4. Engle, Robert & Granger, Clive, 2015. "Co-integration and error correction: Representation, estimation, and testing," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 39(3), pages 106-135.
    5. Spanos,Aris, 1999. "Probability Theory and Statistical Inference," Cambridge Books, Cambridge University Press, number 9780521424080, March.
    6. Bruce E. Hansen, 2001. "The New Econometrics of Structural Change: Dating Breaks in U.S. Labour Productivity," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 117-128, Fall.
    7. Andrews, Donald W K & Ploberger, Werner, 1994. "Optimal Tests When a Nuisance Parameter Is Present Only under the Alternative," Econometrica, Econometric Society, vol. 62(6), pages 1383-1414, November.
    8. Johansen, Soren, 1991. "Estimation and Hypothesis Testing of Cointegration Vectors in Gaussian Vector Autoregressive Models," Econometrica, Econometric Society, vol. 59(6), pages 1551-1580, November.
    9. Jeremy Berkowitz & Lutz Kilian, 2000. "Recent developments in bootstrapping time series," Econometric Reviews, Taylor & Francis Journals, vol. 19(1), pages 1-48.
    10. Jushan Bai & Pierre Perron, 2003. "Computation and analysis of multiple structural change models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 18(1), pages 1-22.
    11. Nelson, Charles R. & Plosser, Charles I., 1982. "Trends and random walks in macroeconmic time series : Some evidence and implications," Journal of Monetary Economics, Elsevier, vol. 10(2), pages 139-162.
    12. Elena Andreou & Aris Spanos, 2003. "Statistical Adequacy and the Testing of Trend Versus Difference Stationarity," Econometric Reviews, Taylor & Francis Journals, vol. 22(3), pages 217-237, January.
    13. Sowell, Fallaw, 1996. "Optimal Tests for Parameter Instability in the Generalized Method of Moments Framework," Econometrica, Econometric Society, vol. 64(5), pages 1085-1107, September.
    14. Andrews, Donald W. K. & Lee, Inpyo & Ploberger, Werner, 1996. "Optimal changepoint tests for normal linear regression," Journal of Econometrics, Elsevier, vol. 70(1), pages 9-38, January.
    15. Perron, Pierre, 1989. "The Great Crash, the Oil Price Shock, and the Unit Root Hypothesis," Econometrica, Econometric Society, vol. 57(6), pages 1361-1401, November.
    16. Banerjee, Anindya & Lumsdaine, Robin L & Stock, James H, 1992. "Recursive and Sequential Tests of the Unit-Root and Trend-Break Hypotheses: Theory and International Evidence," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(3), pages 271-287, July.
    17. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    18. Hansen, Bruce E., 1992. "Testing for parameter instability in linear models," Journal of Policy Modeling, Elsevier, vol. 14(4), pages 517-533, August.
    19. Granger, C. W. J. & Newbold, P., 1974. "Spurious regressions in econometrics," Journal of Econometrics, Elsevier, vol. 2(2), pages 111-120, July.
    20. Vinod, H. D., 2004. "Ranking mutual funds using unconventional utility theory and stochastic dominance," Journal of Empirical Finance, Elsevier, vol. 11(3), pages 353-377, June.
    21. Spanos,Aris, 1986. "Statistical Foundations of Econometric Modelling," Cambridge Books, Cambridge University Press, number 9780521269124, March.
    22. Hansen, Bruce E., 2000. "Testing for structural change in conditional models," Journal of Econometrics, Elsevier, vol. 97(1), pages 93-115, July.
    23. Phillips, P C B, 1987. "Time Series Regression with a Unit Root," Econometrica, Econometric Society, vol. 55(2), pages 277-301, March.
    24. Pierre Perron, 2005. "Dealing with Structural Breaks," Boston University - Department of Economics - Working Papers Series WP2005-017, Boston University - Department of Economics.
    25. Ploberger, Werner & Kramer, Walter & Kontrus, Karl, 1989. "A new test for structural stability in the linear regression model," Journal of Econometrics, Elsevier, vol. 40(2), pages 307-318, February.
    Full references (including those not matched with items on IDEAS)

    More about this item


    Neural networks; resampling; model selection;

    JEL classification:

    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sce:scecfa:497. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F. Baum). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.