IDEAS home Printed from https://ideas.repec.org/p/tiu/tiucen/b8c2217c-f02c-4786-b704-77587495aec8.html
   My bibliography  Save this paper

A Methodology for Fitting and Validating Metamodels in Simulation

Author

Listed:
  • Kleijnen, J.P.C.

    (Tilburg University, Center For Economic Research)

  • Sargent, R.

Abstract

This expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was b
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Kleijnen, J.P.C. & Sargent, R., 1997. "A Methodology for Fitting and Validating Metamodels in Simulation," Discussion Paper 1997-116, Tilburg University, Center for Economic Research.
    • Handle: RePEc:tiu:tiucen:b8c2217c-f02c-4786-b704-77587495aec8
    as

    Download full text from publisher

    File URL: https://pure.uvt.nl/ws/portalfiles/portal/527880/116.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Kleijnen, J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models : Regression analysis and statistical design of experiments," Other publications TiSEM 87ee6ee0-592c-4204-ac50-6, Tilburg University, School of Economics and Management.
    2. Kleijnen, Jack P. C., 1995. "Verification and validation of simulation models," European Journal of Operational Research, Elsevier, vol. 82(1), pages 145-162, April.
    3. Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
    4. Kleijnen, Jack P. C. & Standridge, Charles R., 1988. "Experimental design and regression analysis in simulation: An FMS case study," European Journal of Operational Research, Elsevier, vol. 33(3), pages 257-261, February.
    5. van Ham, G. & Rotmans, J. & Kleijnen, J.P.C., 1992. "Techniques for sensitivity analysis of simulation models : A case study of the CO2 greenhouse effect," Other publications TiSEM 71317a03-3399-4554-83cb-4, Tilburg University, School of Economics and Management.
    6. Kleijnen, J.P.C., 1995. "Statistical validation of simulation models : A case study," Discussion Paper 1995-42, Tilburg University, Center for Economic Research.
    7. Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
    8. Kleijnen, Jack P. C., 1995. "Statistical validation of simulation models," European Journal of Operational Research, Elsevier, vol. 87(1), pages 21-34, November.
    9. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    10. Kleijnen, J.P.C. & Bettonvil, B.W.M., 1997. "Searching for important factors in simulation models with many factors : Sequential bifurcation," Other publications TiSEM be826993-22f9-4cb3-89df-3, Tilburg University, School of Economics and Management.
    11. van Groenendaal, W.J.H. & Kleijnen, J.P.C., 1997. "On the assessment of economic risk : Factorial design versus Monte Carlo methods," Other publications TiSEM fd2a2307-0812-4543-8151-7, Tilburg University, School of Economics and Management.
    12. Bettonvil, Bert & Kleijnen, Jack P. C., 1997. "Searching for important factors in simulation models with many factors: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 96(1), pages 180-194, January.
    13. Saltelli, A. & Andres, T. H. & Homma, T., 1995. "Sensitivity analysis of model output. Performance of the iterated fractional factorial design method," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 387-407, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kleijnen, J.P.C., 1997. "Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models," Discussion Paper 1997-52, Tilburg University, Center for Economic Research.
    2. Ilan Halachmi & Yitzhak Simon & Noam Mozes, 2014. "Simulation of the shift from marine netcages to inland recirculating aquaculture systems," Annals of Operations Research, Springer, vol. 219(1), pages 85-99, August.
    3. Kleijnen, J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models : Regression analysis and statistical design of experiments," Other publications TiSEM 87ee6ee0-592c-4204-ac50-6, Tilburg University, School of Economics and Management.
    4. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    5. Kleijnen, J.P.C. & Sanchez, S.M. & Lucas, T.W. & Cioppa, T.M., 2003. "A User's Guide to the Brave New World of Designing Simulation Experiments," Discussion Paper 2003-1, Tilburg University, Center for Economic Research.
    6. Halachmi, I. & Dzidic, A. & Metz, J. H. M. & Speelman, L. & Dijkhuizen, A. A. & Kleijnen, J. P. C., 2001. "Validation of simulation model for robotic milking barn design," European Journal of Operational Research, Elsevier, vol. 134(3), pages 677-688, November.
    7. Batmaz, Inci & Tunali, Semra, 2003. "Small response surface designs for metamodel estimation," European Journal of Operational Research, Elsevier, vol. 145(2), pages 455-470, March.
    8. Van Groenendaal, Willem J. H. & Kleijnen, Jack P. C., 2002. "Deterministic versus stochastic sensitivity analysis in investment problems: An environmental case study," European Journal of Operational Research, Elsevier, vol. 141(1), pages 8-20, August.
    9. Kleijnen, J. P. C., 2001. "Ethical issues in modeling: Some reflections," European Journal of Operational Research, Elsevier, vol. 130(1), pages 223-230, April.
    10. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Other publications TiSEM d8c37ad3-f9a5-4824-986d-2, Tilburg University, School of Economics and Management.
    11. Shi, Wen & Kleijnen, J.P.C., 2017. "Testing the Assumptions of Sequential Bifurcation for Factor Screening (revision of CentER DP 2015-034)," Other publications TiSEM 763fd6f8-b618-4b06-a284-5, Tilburg University, School of Economics and Management.
    12. Borgonovo, Emanuele & Plischke, Elmar, 2016. "Sensitivity analysis: A review of recent advances," European Journal of Operational Research, Elsevier, vol. 248(3), pages 869-887.
    13. Kleijnen, J.P.C., 2006. "Regression Models and Experimental Designs : A Tutorial for Simulation Analaysts," Discussion Paper 2006-10, Tilburg University, Center for Economic Research.
    14. Tuffa, Samuel & Treydte, Anna C., 2017. "Modeling Boran cattle populations under climate change and varying carrying capacity," Ecological Modelling, Elsevier, vol. 352(C), pages 113-127.
    15. Xuefei Lu & Alessandro Rudi & Emanuele Borgonovo & Lorenzo Rosasco, 2020. "Faster Kriging: Facing High-Dimensional Simulators," Operations Research, INFORMS, vol. 68(1), pages 233-249, January.
    16. Shi, Wen & Shang, Jennifer & Liu, Zhixue & Zuo, Xiaolu, 2014. "Optimal design of the auto parts supply chain for JIT operations: Sequential bifurcation factor screening and multi-response surface methodology," European Journal of Operational Research, Elsevier, vol. 236(2), pages 664-676.
    17. Kleijnen, J.P.C., 2009. "Sensitivity Analysis of Simulation Models," Discussion Paper 2009-11, Tilburg University, Center for Economic Research.
    18. Kleijnen, J.P.C., 1999. "Validation of Models : Statistical Techniques and Data Availability," Discussion Paper 1999-104, Tilburg University, Center for Economic Research.
    19. Legato, Pasquale & Mazza, Rina M., 2001. "Berth planning and resources optimisation at a container terminal via discrete event simulation," European Journal of Operational Research, Elsevier, vol. 133(3), pages 537-547, September.
    20. Shi, Wen & Kleijnen, Jack P.C. & Liu, Zhixue, 2014. "Factor screening for simulation with multiple responses: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 237(1), pages 136-147.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tiu:tiucen:b8c2217c-f02c-4786-b704-77587495aec8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Richard Broekman (email available below). General contact details of provider: http://center.uvt.nl .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.