A methodology for fitting and validating metamodels in simulation
AbstractThis expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.
Bibliographic InfoArticle provided by Elsevier in its journal European Journal of Operational Research.
Volume (Year): 120 (2000)
Issue (Month): 1 (January)
Contact details of provider:
Web page: http://www.elsevier.com/locate/eor
Other versions of this item:
- Kleijnen, J.P.C. & Sargent, R., 1997. "A Methodology for Fitting and Validating Metamodels in Simulation," Discussion Paper 1997-116, Tilburg University, Center for Economic Research.
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Kleijnen, J.P.C., 1992.
"Regression metamodels for simulation with common random numbers: Comparison of validation tests and confidence intervals,"
Open Access publications from Tilburg University
urn:nbn:nl:ui:12-369802, Tilburg University.
- Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
- Kleijnen, J.P.C., 1992.
"Verification and validation of simulation models,"
542, Tilburg University, Faculty of Economics and Business Administration.
- Kleijnen, J.P.C., 1995. "Case study: Statistical validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369796, Tilburg University.
- Tunali, S. & Batmaz, I., 2003. "A metamodeling methodology involving both qualitative and quantitative input factors," European Journal of Operational Research, Elsevier, vol. 150(2), pages 437-450, October.
- Husslage, B.G.M. & Dam, E.R. van & Hertog, D. den & Stehouwer, H.P. & Stinstra, E., 2003. "Coordination of Coupled Black Box Simulations in the Construction of Metamodels," Discussion Paper 2003-2, Tilburg University, Center for Economic Research.
- Wise, Russell M. & Cacho, Oscar J., 2006. "Optimal Land-Use Decisions in the Presence of Carbon Payments and Fertilizer Subsidies: An Indonesian Case Study," 2006 Annual Meeting, August 12-18, 2006, Queensland, Australia 25356, International Association of Agricultural Economists.
- Batmaz, Inci & Tunali, Semra, 2003. "Small response surface designs for metamodel estimation," European Journal of Operational Research, Elsevier, vol. 145(2), pages 455-470, March.
- Graham, Tennille, 2005. "On the Road to Better Management: An investigation into the benefits of managing the impacts of dryland salinity on roads," 2005 Conference (49th), February 9-11, 2005, Coff's Harbour, Australia 137921, Australian Agricultural and Resource Economics Society.
- Noguera, Jose H. & Watson, Edward F., 2006. "Response surface analysis of a multi-product batch processing facility using a simulation metamodel," International Journal of Production Economics, Elsevier, vol. 102(2), pages 333-343, August.
- Kleijnen, Jack P.C. & Mehdad, E., 2012. "Kriging in Multi-response Simulation, including a Monte Carlo Laboratory," Discussion Paper 2012-039, Tilburg University, Center for Economic Research.
- Poropudas, Jirka & Virtanen, Kai, 2011. "Simulation metamodeling with dynamic Bayesian networks," European Journal of Operational Research, Elsevier, vol. 214(3), pages 644-655, November.
- Marrel, Amandine & Iooss, Bertrand & Van Dorpe, François & Volkova, Elena, 2008. "An efficient methodology for modeling complex computer codes with Gaussian processes," Computational Statistics & Data Analysis, Elsevier, vol. 52(10), pages 4731-4744, June.
- Reis dos Santos, Pedro M. & Isabel Reis dos Santos, M., 2009. "Using subsystem linear regression metamodels in stochastic simulation," European Journal of Operational Research, Elsevier, vol. 196(3), pages 1031-1040, August.
- Ekren, Orhan & Ekren, Banu Y. & Ozerdem, Baris, 2009. "Break-even analysis and size optimization of a PV/wind hybrid energy conversion system with battery storage - A case study," Applied Energy, Elsevier, vol. 86(7-8), pages 1043-1054, July.
- Vonk Noordegraaf, Antonie & Nielen, Mirjam & Kleijnen, Jack P. C., 2003. "Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control," European Journal of Operational Research, Elsevier, vol. 146(3), pages 433-443, May.
- van der Gaag, Monique A. & Vos, Fred & Saatkamp, Helmut W. & van Boven, Michiel & van Beek, Paul & Huirne, Ruud B. M., 2004. "A state-transition simulation model for the spread of Salmonella in the pork supply chain," European Journal of Operational Research, Elsevier, vol. 156(3), pages 782-798, August.
- Galelli, S. & Gandolfi, C. & Soncini-Sessa, R. & Agostani, D., 2010. "Building a metamodel of an irrigation district distributed-parameter model," Agricultural Water Management, Elsevier, vol. 97(2), pages 187-200, February.
- Strang, Kenneth David, 2012. "Importance of verifying queue model assumptions before planning with simulation software," European Journal of Operational Research, Elsevier, vol. 218(2), pages 493-504.
- Durieux, Severine & Pierreval, Henri, 2004. "Regression metamodeling for the design of automated manufacturing system composed of parallel machines sharing a material handling resource," International Journal of Production Economics, Elsevier, vol. 89(1), pages 21-30, May.
- Ekren, Orhan & Ekren, Banu Yetkin, 2008. "Size optimization of a PV/wind hybrid energy conversion system with battery storage using response surface methodology," Applied Energy, Elsevier, vol. 85(11), pages 1086-1101, November.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wendy Shamier).
If references are entirely missing, you can add them using this form.