A methodology for fitting and validating metamodels in simulation
This expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.
(This abstract was borrowed from another version of this item.)
If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.
References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
- Diebold, Francis X & Mariano, Roberto S, 1995.
"Comparing Predictive Accuracy,"
Journal of Business & Economic Statistics,
American Statistical Association, vol. 13(3), pages 253-263, July.
- Francis X. Diebold & Robert S. Mariano, 1994. "Comparing Predictive Accuracy," NBER Technical Working Papers 0169, National Bureau of Economic Research, Inc.
- Tom Doan, "undated". "DMARIANO: RATS procedure to compute Diebold-Mariano Forecast Comparison Test," Statistical Software Components RTS00055, Boston College Department of Economics.
- Kleijnen, J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models : Regression analysis and statistical design of experiments," Discussion Paper 1995-4, Tilburg University, Center for Economic Research.
- Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
- Kleijnen, Jack P. C. & Standridge, Charles R., 1988.
"Experimental design and regression analysis in simulation: An FMS case study,"
European Journal of Operational Research,
Elsevier, vol. 33(3), pages 257-261, February.
- Kleijnen, J.P.C. & Standridge, C.R., 1986. "Experimental design and regression analysis in simulation : An FMS case study," Research Memorandum FEW 212, Tilburg University, School of Economics and Management.
- van Groenendaal, W.J.H. & Kleijnen, J.P.C., 1997. "On the assessment of economic risk : Factorial design versus Monte Carlo methods," Other publications TiSEM fd2a2307-0812-4543-8151-7, Tilburg University, School of Economics and Management.
- Kleijnen, J.P.C., 1992.
"Verification and validation of simulation models,"
FEW 542, Tilburg University, School of Economics and Management.
- Saltelli, A. & Andres, T. H. & Homma, T., 1995. "Sensitivity analysis of model output. Performance of the iterated fractional factorial design method," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 387-407, October.
- Bettonvil, Bert & Kleijnen, Jack P. C., 1997. "Searching for important factors in simulation models with many factors: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 96(1), pages 180-194, January.
- van Ham, G. & Rotmans, J. & Kleijnen, J.P.C., 1992. "Techniques for sensitivity analysis of simulation models : A case study of the CO2 greenhouse effect," Other publications TiSEM 71317a03-3399-4554-83cb-4, Tilburg University, School of Economics and Management.
When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:120:y:2000:i:1:p:14-29. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu)
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If references are entirely missing, you can add them using this form.
If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.
Please note that corrections may take a couple of weeks to filter through the various RePEc services.