Advanced Search
MyIDEAS: Login

Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models


Author Info

  • Kleijnen, J.P.C.

    (Tilburg University, Center for Economic Research)


This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL:
Our checks indicate that this address may not be valid because: 404 Not Found. If this is indeed the case, please notify (Richard Broekman)
Download Restriction: no

Bibliographic Info

Paper provided by Tilburg University, Center for Economic Research in its series Discussion Paper with number 1997-52.

as in new window
Date of creation: 1997
Date of revision:
Handle: RePEc:dgr:kubcen:199752

Contact details of provider:
Web page:

Related research

Keywords: least squares; distribution-free; non-parametric; stopping rule; run-length; Von Neumann; median; seed; likelihood ratio;


References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Kleijnen, Jack P. C. & Standridge, Charles R., 1988. "Experimental design and regression analysis in simulation: An FMS case study," European Journal of Operational Research, Elsevier, vol. 33(3), pages 257-261, February.
  2. Bettonvil, Bert & Kleijnen, Jack P. C., 1997. "Searching for important factors in simulation models with many factors: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 96(1), pages 180-194, January.
  3. Sherif, Yosef S. & Boice, Bruce A., 1994. "Optimization by pattern search," European Journal of Operational Research, Elsevier, vol. 78(3), pages 277-303, November.
  4. Kleijnen, J.P.C., 1975. "A comment on Blanning's metamodel for sensitivity analysis: The regression metamodel in simulation," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369888, Tilburg University.
  5. Kleijnen, J.P.C., 1995. "Case study: Statistical validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369796, Tilburg University.
  6. Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
  7. Kleijnen, J.P.C. & Groenendaal, W.J.H. van, 1995. "Two-stage versus sequential sample-size determination in regression analysis of simulation experiments," Open Access publications from Tilburg University urn:nbn:nl:ui:12-80492, Tilburg University.
  8. Dagenais, M.G. & Dufour, J.M., 1993. "Pitfalls of Rescalling Regression Models with Box-Cox Transformations," Cahiers de recherche 9333a, Centre interuniversitaire de recherche en économie quantitative, CIREQ.
  9. Kleijnen, J.P.C. & Karremans, P.C.A. & Oortwijn , W.K. & Groenendaal, W.J.H. van, 1987. "Jackknifing estimated weighted least squares: JEWLS," Open Access publications from Tilburg University urn:nbn:nl:ui:12-365635, Tilburg University.
  10. Kleijnen, J.P.C., 1995. "Verification and validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369797, Tilburg University.
  11. Standridge, C. & Kleijnen, J.P.C., 1988. "Experimental design and regression analysis: An FMS case study," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369815, Tilburg University.
  12. Kleijnen, J.P.C. & Kloppenburg, G. & Meeuwsen, F., 1985. "Testing the mean of an asymmetric population: Johnson's modified t test revisited," Research Memorandum 183, Tilburg University, Faculty of Economics and Business Administration.
  13. Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
  14. Kleijnen, J.P.C. & Groenendaal, W.J.H. van, 1992. "Simulation: A statistical perspective," Open Access publications from Tilburg University urn:nbn:nl:ui:12-388278, Tilburg University.
  15. Kleijnen, Jack P. C., 1995. "Statistical validation of simulation models," European Journal of Operational Research, Elsevier, vol. 87(1), pages 21-34, November.
  16. Kleijnen, J.P.C., 2009. "Sensitivity Analysis of Simulation Models," Discussion Paper 2009-11, Tilburg University, Center for Economic Research.
  17. Kleijnen, J.P.C., 1993. "Simulation and optimization in production planning: a case study," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369798, Tilburg University.
  18. Kleijnen , J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models: Regression analysis and statistical design of experiments," Open Access publications from Tilburg University urn:nbn:nl:ui:12-365599, Tilburg University.
  19. Saltelli, A. & Andres, T. H. & Homma, T., 1995. "Sensitivity analysis of model output. Performance of the iterated fractional factorial design method," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 387-407, October.
Full references (including those not matched with items on IDEAS)


Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Grazia Vicario & Daniele Romano, 2001. "Factorial experiments for sequential process," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(1-2), pages 191-204.
  2. Kleijnen, J.P.C., 2004. "An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis," Discussion Paper 2004-16, Tilburg University, Center for Economic Research.
  3. Matteo Richiardi, 2004. "The Promises and Perils of Agent-Based Computational Economics," Computational Economics 0401001, EconWPA.
  4. Kleijnen, J.P.C., 2001. "Experimental Design for Sensitivity Analysis of Simulation Models," Discussion Paper 2001-15, Tilburg University, Center for Economic Research.
  5. Giorgio Fagiolo & Paul Windrum & Alessio Moneta, 2006. "Empirical Validation of Agent Based Models: A Critical Survey," LEM Papers Series 2006/14, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
  6. Gaury, E.G.A. & Kleijnen, J.P.C. & Pierreval, H., 1997. "Configuring a pull production control strategy through a generic model," Discussion Paper 1997-101, Tilburg University, Center for Economic Research.
  7. Gaury, E.G.A. & Kleijnen, J.P.C. & Pierreval, H., 1998. "Customized Pull Systems for Single-Product Flow Lines," Discussion Paper 1998-117, Tilburg University, Center for Economic Research.
  8. Kurt DeMaagd & Scott Moore, 2007. "Computational modeling of city formation," Computational Economics, Society for Computational Economics, vol. 30(1), pages 41-56, August.
  9. Beers, W.C.M. van & Kleijnen, J.P.C., 2001. "Kriging for Interpolation in Random Simulation," Discussion Paper 2001-74, Tilburg University, Center for Economic Research.
  10. Pasquale Cirillo & Carlo Bianchi & Mauro Gallegati & Pietro Vagliasindi, 2006. "Validating and Calibrating Agent-based Models: a Case Study," Computing in Economics and Finance 2006 277, Society for Computational Economics.
  11. Kleijnen, J.P.C., 2006. "Generalized Response Surface Methodology: A New Metaheuristic," Discussion Paper 2006-77, Tilburg University, Center for Economic Research.


This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.


Access and download statistics


When requesting a correction, please mention this item's handle: RePEc:dgr:kubcen:199752. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Richard Broekman).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.