A methodology for fitting and validating metamodels in simulation
AbstractThis expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.
Bibliographic InfoArticle provided by Elsevier in its journal European Journal of Operational Research.
Volume (Year): 120 (2000)
Issue (Month): 1 (January)
Contact details of provider:
Web page: http://www.elsevier.com/locate/eor
Other versions of this item:
- Kleijnen, J.P.C. & Sargent, R., 1997. "A Methodology for Fitting and Validating Metamodels in Simulation," Discussion Paper 1997-116, Tilburg University, Center for Economic Research.
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Kleijnen, J.P.C., 1992.
"Verification and validation of simulation models,"
542, Tilburg University, Faculty of Economics and Business Administration.
- Jack P. C. Kleijnen, 1992.
"Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals,"
INFORMS, vol. 38(8), pages 1164-1185, August.
- Kleijnen, J.P.C., 1992. "Regression metamodels for simulation with common random numbers: Comparison of validation tests and confidence intervals," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369802, Tilburg University.
- Kleijnen, J.P.C., 1995. "Case study: Statistical validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369796, Tilburg University.
- Husslage, B.G.M. & Dam, E.R. van & Hertog, D. den & Stehouwer, H.P. & Stinstra, E., 2003. "Coordination of Coupled Black Box Simulations in the Construction of Metamodels," Discussion Paper 2003-2, Tilburg University, Center for Economic Research.
- Durieux, Severine & Pierreval, Henri, 2004. "Regression metamodeling for the design of automated manufacturing system composed of parallel machines sharing a material handling resource," International Journal of Production Economics, Elsevier, vol. 89(1), pages 21-30, May.
- Graham, Tennille, 2005. "On the Road to Better Management: An investigation into the benefits of managing the impacts of dryland salinity on roads," 2005 Conference (49th), February 9-11, 2005, Coff's Harbour, Australia 137921, Australian Agricultural and Resource Economics Society.
- Marjolijn Haasnoot & Hans Middelkoop & Astrid Offermans & Eelco Beek & Willem Deursen, 2012. "Exploring pathways for sustainable water management in river deltas in a changing environment," Climatic Change, Springer, vol. 115(3), pages 795-819, December.
- Vonk Noordegraaf, Antonie & Nielen, Mirjam & Kleijnen, Jack P. C., 2003. "Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control," European Journal of Operational Research, Elsevier, vol. 146(3), pages 433-443, May.
- Galelli, S. & Gandolfi, C. & Soncini-Sessa, R. & Agostani, D., 2010. "Building a metamodel of an irrigation district distributed-parameter model," Agricultural Water Management, Elsevier, vol. 97(2), pages 187-200, February.
- Poropudas, Jirka & Virtanen, Kai, 2011. "Simulation metamodeling with dynamic Bayesian networks," European Journal of Operational Research, Elsevier, vol. 214(3), pages 644-655, November.
- Ekren, Orhan & Ekren, Banu Yetkin, 2008. "Size optimization of a PV/wind hybrid energy conversion system with battery storage using response surface methodology," Applied Energy, Elsevier, vol. 85(11), pages 1086-1101, November.
- Reis dos Santos, Pedro M. & Isabel Reis dos Santos, M., 2009. "Using subsystem linear regression metamodels in stochastic simulation," European Journal of Operational Research, Elsevier, vol. 196(3), pages 1031-1040, August.
- Noguera, Jose H. & Watson, Edward F., 2006. "Response surface analysis of a multi-product batch processing facility using a simulation metamodel," International Journal of Production Economics, Elsevier, vol. 102(2), pages 333-343, August.
- Strang, Kenneth David, 2012. "Importance of verifying queue model assumptions before planning with simulation software," European Journal of Operational Research, Elsevier, vol. 218(2), pages 493-504.
- Batmaz, Inci & Tunali, Semra, 2003. "Small response surface designs for metamodel estimation," European Journal of Operational Research, Elsevier, vol. 145(2), pages 455-470, March.
- Kleijnen, Jack P.C. & Mehdad, E., 2012. "Kriging in Multi-response Simulation, including a Monte Carlo Laboratory (Replaced by 2014-012)," Discussion Paper 2012-039, Tilburg University, Center for Economic Research.
- Ekren, Orhan & Ekren, Banu Y. & Ozerdem, Baris, 2009. "Break-even analysis and size optimization of a PV/wind hybrid energy conversion system with battery storage - A case study," Applied Energy, Elsevier, vol. 86(7-8), pages 1043-1054, July.
- Wise, Russell M. & Cacho, Oscar J., 2006. "Optimal Land-Use Decisions in the Presence of Carbon Payments and Fertilizer Subsidies: An Indonesian Case Study," 2006 Annual Meeting, August 12-18, 2006, Queensland, Australia 25356, International Association of Agricultural Economists.
- Tunali, S. & Batmaz, I., 2003. "A metamodeling methodology involving both qualitative and quantitative input factors," European Journal of Operational Research, Elsevier, vol. 150(2), pages 437-450, October.
- Shi, Wen & Shang, Jennifer & Liu, Zhixue & Zuo, Xiaolu, 2014. "Optimal design of the auto parts supply chain for JIT operations: Sequential bifurcation factor screening and multi-response surface methodology," European Journal of Operational Research, Elsevier, vol. 236(2), pages 664-676.
- van der Gaag, Monique A. & Vos, Fred & Saatkamp, Helmut W. & van Boven, Michiel & van Beek, Paul & Huirne, Ruud B. M., 2004. "A state-transition simulation model for the spread of Salmonella in the pork supply chain," European Journal of Operational Research, Elsevier, vol. 156(3), pages 782-798, August.
- Marrel, Amandine & Iooss, Bertrand & Van Dorpe, François & Volkova, Elena, 2008. "An efficient methodology for modeling complex computer codes with Gaussian processes," Computational Statistics & Data Analysis, Elsevier, vol. 52(10), pages 4731-4744, June.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei).
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If references are entirely missing, you can add them using this form.
If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.
Please note that corrections may take a couple of weeks to filter through the various RePEc services.