IDEAS home Printed from https://ideas.repec.org/a/inm/orijoc/v34y2022i1p607-621.html
   My bibliography  Save this article

Statistical Tests for Cross-Validation of Kriging Models

Author

Listed:
  • Jack P. C. Kleijnen

    (Department of Management, Tilburg School of Economics and Management, Tilburg University, 5000 LE Tilburg, Netherlands)

  • Wim C. M. van Beers

    (Department of Management, Tilburg School of Economics and Management, Tilburg University, 5000 LE Tilburg, Netherlands)

Abstract

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model.

Suggested Citation

  • Jack P. C. Kleijnen & Wim C. M. van Beers, 2022. "Statistical Tests for Cross-Validation of Kriging Models," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 607-621, January.
    • Handle: RePEc:inm:orijoc:v:34:y:2022:i:1:p:607-621
    DOI: 10.1287/ijoc.2021.1072
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/ijoc.2021.1072
    Download Restriction: no
    File URL: https://libkey.io/10.1287/ijoc.2021.1072?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. D den Hertog & J P C Kleijnen & A Y D Siem, 2006. "The correct Kriging variance estimated by bootstrapping," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 57(4), pages 400-409, April.
    2. Kleijnen, Jack P. C., 1983. "Cross-validation using the t statistic," European Journal of Operational Research, Elsevier, vol. 13(2), pages 133-141, June.
    3. Roustant, Olivier & Ginsbourger, David & Deville, Yves, 2012. "DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 51(i01).
    4. Erickson, Collin B. & Ankenman, Bruce E. & Sanchez, Susan M., 2018. "Comparison of Gaussian process modeling software," European Journal of Operational Research, Elsevier, vol. 266(1), pages 179-192.
    5. Bachoc, François & Lagnoux, Agnès & Nguyen, Thi Mong Ngoc, 2017. "Cross-validation estimation of covariance parameters under fixed-domain asymptotics," Journal of Multivariate Analysis, Elsevier, vol. 160(C), pages 42-67.
    6. Yujing Lin & Barry L. Nelson & Linda Pei, 2019. "Virtual Statistics in Simulation via k Nearest Neighbors," INFORMS Journal on Computing, INFORMS, vol. 31(3), pages 576-592, July.
    7. Gramacy, Robert B., 2016. "laGP: Large-Scale Spatial Modeling via Local Approximate Gaussian Processes in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 72(i01).
    8. Bradley Efron, 2015. "Frequentist accuracy of Bayesian estimates," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(3), pages 617-646, June.
    9. Guangxin Jiang & L. Jeff Hong & Barry L. Nelson, 2020. "Online Risk Monitoring Using Offline Simulation," INFORMS Journal on Computing, INFORMS, vol. 32(2), pages 356-375, April.
    10. Peter Salemi & Jeremy Staum & Barry L. Nelson, 2019. "Generalized Integrated Brownian Fields for Simulation Metamodeling," Operations Research, INFORMS, vol. 67(3), pages 874-891, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pata, Ugur Korkut & Kartal, Mustafa Tevfik & Erdogan, Sinan & Sarkodie, Samuel Asumadu, 2023. "The role of renewable and nuclear energy R&D expenditures and income on environmental quality in Germany: Scrutinizing the EKC and LCC hypotheses with smooth structural changes," Applied Energy, Elsevier, vol. 342(C).
    2. Zhang, Mengling & Jiao, Zihao & Ran, Lun & Zhang, Yuli, 2023. "Optimal energy and reserve scheduling in a renewable-dominant power system," Omega, Elsevier, vol. 118(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xuefei Lu & Alessandro Rudi & Emanuele Borgonovo & Lorenzo Rosasco, 2020. "Faster Kriging: Facing High-Dimensional Simulators," Operations Research, INFORMS, vol. 68(1), pages 233-249, January.
    2. Kleijnen, Jack & van Beers, W.C.M., 2019. "Statistical Tests for Cross-Validation of Kriging Models," Other publications TiSEM 35fba511-2931-47d5-a9ba-3, Tilburg University, School of Economics and Management.
    3. Kleijnen, Jack P.C. & Mehdad, Ehsan, 2014. "Multivariate versus univariate Kriging metamodels for multi-response simulation models," European Journal of Operational Research, Elsevier, vol. 236(2), pages 573-582.
    4. Kleijnen, Jack & van Nieuwenhuyse, I. & van Beers, W.C.M., 2022. "Constrained Optimization in Simulation : Efficient Global Optimization and Karush-Kuhn-Tucker Conditions (revision of 2021-031)," Other publications TiSEM 31a06a3b-dfc4-4431-a141-5, Tilburg University, School of Economics and Management.
    5. Kleijnen, Jack P.C., 2013. "Simulation-Optimization via Kriging and Bootstrapping : A Survey (Revision of CentER DP 2011-064)," Other publications TiSEM 6ac4e049-ad86-447f-aeec-a, Tilburg University, School of Economics and Management.
    6. Kleijnen, Jack P.C. & Mehdad, E., 2014. "Multivariate Versus Univariate Kriging Metamodels for Multi-Response Simulation Models (Revision of 2012-039)," Discussion Paper 2014-012, Tilburg University, Center for Economic Research.
    7. Mostafa Reisi Gahrooei & Hao Yan & Kamran Paynabar, 2020. "Comments on: On Active Learning Methods for Manifold Data," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 29(1), pages 38-41, March.
    8. Hans Olav Vogt Myklebust & Jo Eidsvik & Iver Bakken Sperstad & Debarun Bhattacharjya, 2020. "Value of Information Analysis for Complex Simulator Models: Application to Wind Farm Maintenance," Decision Analysis, INFORMS, vol. 17(2), pages 134-153, June.
    9. Bachoc, François & Bevilacqua, Moreno & Velandia, Daira, 2019. "Composite likelihood estimation for a Gaussian process under fixed domain asymptotics," Journal of Multivariate Analysis, Elsevier, vol. 174(C).
    10. Ehsan Mehdad & Jack P. C. Kleijnen, 2018. "Efficient global optimisation for black-box simulation via sequential intrinsic Kriging," Journal of the Operational Research Society, Taylor & Francis Journals, vol. 69(11), pages 1725-1737, November.
    11. Diariétou Sambakhé & Lauriane Rouan & Jean-Noël Bacro & Eric Gozé, 2019. "Conditional optimization of a noisy function using a kriging metamodel," Journal of Global Optimization, Springer, vol. 73(3), pages 615-636, March.
    12. Zhang, Wei & (Ato) Xu, Wangtu, 2017. "Simulation-based robust optimization for the schedule of single-direction bus transit route: The design of experiment," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 106(C), pages 203-230.
    13. Biewen, Martin & Kugler, Philipp, 2021. "Two-stage least squares random forests with an application to Angrist and Evans (1998)," Economics Letters, Elsevier, vol. 204(C).
    14. Matthew Reimherr & Xiao‐Li Meng & Dan L. Nicolae, 2021. "Prior sample size extensions for assessing prior impact and prior‐likelihood discordance," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(3), pages 413-437, July.
    15. Olgun Aydin & Bartłomiej Igliński & Krzysztof Krukowski & Marek Siemiński, 2022. "Analyzing Wind Energy Potential Using Efficient Global Optimization: A Case Study for the City Gdańsk in Poland," Energies, MDPI, vol. 15(9), pages 1-22, April.
    16. Mehdad, E. & Kleijnen, Jack P.C., 2014. "Classic Kriging versus Kriging with Bootstrapping or Conditional Simulation : Classic Kriging's Robust Confidence Intervals and Optimization (Revised version of CentER DP 2013-038)," Other publications TiSEM 4915047b-afe4-4fc7-8a1c-4, Tilburg University, School of Economics and Management.
    17. Dellino, G. & Lino, P. & Meloni, C. & Rizzo, A., 2009. "Kriging metamodel management in the design optimization of a CNG injection system," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 79(8), pages 2345-2360.
    18. Franks Alexander M. & D’Amour Alexander & Cervone Daniel & Bornn Luke, 2016. "Meta-analytics: tools for understanding the statistical properties of sports metrics," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 12(4), pages 151-165, December.
    19. Erickson, Collin B. & Ankenman, Bruce E. & Sanchez, Susan M., 2018. "Comparison of Gaussian process modeling software," European Journal of Operational Research, Elsevier, vol. 266(1), pages 179-192.
    20. van Beers, Wim C.M. & Kleijnen, Jack P.C., 2008. "Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping," European Journal of Operational Research, Elsevier, vol. 186(3), pages 1099-1113, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orijoc:v:34:y:2022:i:1:p:607-621. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.