A Methodology for Fitting and Validating Metamodels in Simulation
AbstractThis expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Tilburg University, Center for Economic Research in its series Discussion Paper with number 1997-116.
Date of creation: 1997
Date of revision:
Contact details of provider:
Web page: http://center.uvt.nl
Simulation; approximation; response surface; modelling; regression;
Other versions of this item:
- Kleijnen, Jack P. C. & Sargent, Robert G., 2000. "A methodology for fitting and validating metamodels in simulation," European Journal of Operational Research, Elsevier, vol. 120(1), pages 14-29, January.
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Kleijnen, J.P.C., 1995.
"Verification and validation of simulation models,"
Open Access publications from Tilburg University
urn:nbn:nl:ui:12-369797, Tilburg University.
- Kleijnen, J.P.C., 1992.
"Regression metamodels for simulation with common random numbers: Comparison of validation tests and confidence intervals,"
Open Access publications from Tilburg University
urn:nbn:nl:ui:12-369802, Tilburg University.
- Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
- Kleijnen, J.P.C., 1995. "Case study: Statistical validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369796, Tilburg University.
- Vonk Noordegraaf, Antonie & Nielen, Mirjam & Kleijnen, Jack P. C., 2003. "Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control," European Journal of Operational Research, Elsevier, vol. 146(3), pages 433-443, May.
- Husslage, B.G.M. & Dam, E.R. van & Hertog, D. den & Stehouwer, H.P. & Stinstra, E., 2003. "Coordination of Coupled Black Box Simulations in the Construction of Metamodels," Discussion Paper 2003-2, Tilburg University, Center for Economic Research.
- Graham, Tennille, 2005. "On the Road to Better Management: An investigation into the benefits of managing the impacts of dryland salinity on roads," 2005 Conference (49th), February 9-11, 2005, Coff's Harbour, Australia 137921, Australian Agricultural and Resource Economics Society.
- Marrel, Amandine & Iooss, Bertrand & Van Dorpe, François & Volkova, Elena, 2008. "An efficient methodology for modeling complex computer codes with Gaussian processes," Computational Statistics & Data Analysis, Elsevier, vol. 52(10), pages 4731-4744, June.
- Galelli, S. & Gandolfi, C. & Soncini-Sessa, R. & Agostani, D., 2010. "Building a metamodel of an irrigation district distributed-parameter model," Agricultural Water Management, Elsevier, vol. 97(2), pages 187-200, February.
- Ekren, Orhan & Ekren, Banu Y. & Ozerdem, Baris, 2009. "Break-even analysis and size optimization of a PV/wind hybrid energy conversion system with battery storage - A case study," Applied Energy, Elsevier, vol. 86(7-8), pages 1043-1054, July.
- Strang, Kenneth David, 2012. "Importance of verifying queue model assumptions before planning with simulation software," European Journal of Operational Research, Elsevier, vol. 218(2), pages 493-504.
- Noguera, Jose H. & Watson, Edward F., 2006. "Response surface analysis of a multi-product batch processing facility using a simulation metamodel," International Journal of Production Economics, Elsevier, vol. 102(2), pages 333-343, August.
- Wise, Russell M. & Cacho, Oscar J., 2006. "Optimal Land-Use Decisions in the Presence of Carbon Payments and Fertilizer Subsidies: An Indonesian Case Study," 2006 Annual Meeting, August 12-18, 2006, Queensland, Australia 25356, International Association of Agricultural Economists.
- van der Gaag, Monique A. & Vos, Fred & Saatkamp, Helmut W. & van Boven, Michiel & van Beek, Paul & Huirne, Ruud B. M., 2004. "A state-transition simulation model for the spread of Salmonella in the pork supply chain," European Journal of Operational Research, Elsevier, vol. 156(3), pages 782-798, August.
- Marjolijn Haasnoot & Hans Middelkoop & Astrid Offermans & Eelco Beek & Willem Deursen, 2012. "Exploring pathways for sustainable water management in river deltas in a changing environment," Climatic Change, Springer, vol. 115(3), pages 795-819, December.
- Tunali, S. & Batmaz, I., 2003. "A metamodeling methodology involving both qualitative and quantitative input factors," European Journal of Operational Research, Elsevier, vol. 150(2), pages 437-450, October.
- Kleijnen, Jack P.C. & Mehdad, E., 2012. "Kriging in Multi-response Simulation, including a Monte Carlo Laboratory (Replaced by 2014-012)," Discussion Paper 2012-039, Tilburg University, Center for Economic Research.
- Batmaz, Inci & Tunali, Semra, 2003. "Small response surface designs for metamodel estimation," European Journal of Operational Research, Elsevier, vol. 145(2), pages 455-470, March.
- Shi, Wen & Shang, Jennifer & Liu, Zhixue & Zuo, Xiaolu, 2014. "Optimal design of the auto parts supply chain for JIT operations: Sequential bifurcation factor screening and multi-response surface methodology," European Journal of Operational Research, Elsevier, vol. 236(2), pages 664-676.
- Ekren, Orhan & Ekren, Banu Yetkin, 2008. "Size optimization of a PV/wind hybrid energy conversion system with battery storage using response surface methodology," Applied Energy, Elsevier, vol. 85(11), pages 1086-1101, November.
- Durieux, Severine & Pierreval, Henri, 2004. "Regression metamodeling for the design of automated manufacturing system composed of parallel machines sharing a material handling resource," International Journal of Production Economics, Elsevier, vol. 89(1), pages 21-30, May.
- Reis dos Santos, Pedro M. & Isabel Reis dos Santos, M., 2009. "Using subsystem linear regression metamodels in stochastic simulation," European Journal of Operational Research, Elsevier, vol. 196(3), pages 1031-1040, August.
- Poropudas, Jirka & Virtanen, Kai, 2011. "Simulation metamodeling with dynamic Bayesian networks," European Journal of Operational Research, Elsevier, vol. 214(3), pages 644-655, November.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Richard Broekman).
If references are entirely missing, you can add them using this form.