Advanced Search
MyIDEAS: Login to save this paper or follow this series

Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models


Author Info

  • Kleijnen, J.P.C.

    (Tilburg University, Center for Economic Research)


This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL:
Our checks indicate that this address may not be valid because: 404 Not Found. If this is indeed the case, please notify (Richard Broekman)
Download Restriction: no

Bibliographic Info

Paper provided by Tilburg University, Center for Economic Research in its series Discussion Paper with number 1997-52.

as in new window
Date of creation: 1997
Date of revision:
Handle: RePEc:dgr:kubcen:199752

Contact details of provider:
Web page:

Related research

Keywords: least squares; distribution-free; non-parametric; stopping rule; run-length; Von Neumann; median; seed; likelihood ratio;


References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Kleijnen, J.P.C., 1975. "A comment on Blanning's metamodel for sensitivity analysis: The regression metamodel in simulation," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369888, Tilburg University.
  2. Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
  3. Kleijnen, J.P.C., 1995. "Case study: Statistical validation of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369796, Tilburg University.
  4. Kleijnen, J.P.C., 1992. "Verification and validation of simulation models," Research Memorandum 542, Tilburg University, Faculty of Economics and Business Administration.
  5. Kleijnen, J.P.C. & Kloppenburg , G.L.J. & Meeuwsen , F.L., 1986. "Testing the mean of an asymmetric population: Johnson's modified t test revisited," Open Access publications from Tilburg University urn:nbn:nl:ui:12-365641, Tilburg University.
  6. Dagenais, M.G. & Dufour, J.M., 1993. "Pitfalls of Rescalling Regression Models with Box-Cox Transformations," Cahiers de recherche 9333a, Centre interuniversitaire de recherche en économie quantitative, CIREQ.
  7. Kleijnen, J.P.C., 1992. "Regression metamodels for simulation with common random numbers: Comparison of validation tests and confidence intervals," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369802, Tilburg University.
  8. Kleijnen, J.P.C. & Groenendaal, W.J.H. van, 1994. "Two-stage versus sequential sample-size determination in regression analysis of simulation experiments," Discussion Paper 1994-28, Tilburg University, Center for Economic Research.
  9. Kleijnen, J.P.C. & Groenendaal, W.J.H. van, 1992. "Simulation: A statistical perspective," Open Access publications from Tilburg University urn:nbn:nl:ui:12-388278, Tilburg University.
  10. Kleijnen, Jack P. C. & Standridge, Charles R., 1988. "Experimental design and regression analysis in simulation: An FMS case study," European Journal of Operational Research, Elsevier, vol. 33(3), pages 257-261, February.
  11. Kleijnen, J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models: Regression analysis and statistical design of experiments," Discussion Paper 1995-4, Tilburg University, Center for Economic Research.
  12. Kleijnen, J.P.C. & Karremans, P.C.A. & Oortwijn , W.K. & Groenendaal, W.J.H. van, 1987. "Jackknifing estimated weighted least squares: JEWLS," Open Access publications from Tilburg University urn:nbn:nl:ui:12-365635, Tilburg University.
  13. Kleijnen, J.P.C., 2009. "Sensitivity Analysis of Simulation Models," Discussion Paper 2009-11, Tilburg University, Center for Economic Research.
  14. Sherif, Yosef S. & Boice, Bruce A., 1994. "Optimization by pattern search," European Journal of Operational Research, Elsevier, vol. 78(3), pages 277-303, November.
  15. Bettonvil, Bert & Kleijnen, Jack P. C., 1997. "Searching for important factors in simulation models with many factors: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 96(1), pages 180-194, January.
  16. Kleijnen, J.P.C., 1993. "Simulation and optimization in production planning: a case study," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369798, Tilburg University.
  17. Standridge, C. & Kleijnen, J.P.C., 1988. "Experimental design and regression analysis: An FMS case study," Open Access publications from Tilburg University urn:nbn:nl:ui:12-369815, Tilburg University.
  18. Kleijnen, Jack P. C., 1995. "Statistical validation of simulation models," European Journal of Operational Research, Elsevier, vol. 87(1), pages 21-34, November.
  19. Saltelli, A. & Andres, T. H. & Homma, T., 1995. "Sensitivity analysis of model output. Performance of the iterated fractional factorial design method," Computational Statistics & Data Analysis, Elsevier, vol. 20(4), pages 387-407, October.
Full references (including those not matched with items on IDEAS)


Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Society for Computational Economics, vol. 30(3), pages 245-264, October.
  2. Beers, W.C.M. van & Kleijnen, J.P.C., 2001. "Kriging for Interpolation in Random Simulation," Discussion Paper 2001-74, Tilburg University, Center for Economic Research.
  3. Giorgio Fagiolo & Paul Windrum & Alessio Moneta, 2006. "Empirical Validation of Agent Based Models: A Critical Survey," LEM Papers Series 2006/14, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
  4. Kleijnen, Jack P. C., 2005. "An overview of the design and analysis of simulation experiments for sensitivity analysis," European Journal of Operational Research, Elsevier, vol. 164(2), pages 287-300, July.
  5. Kurt DeMaagd & Scott Moore, 2007. "Computational modeling of city formation," Computational Economics, Society for Computational Economics, vol. 30(1), pages 41-56, August.
  6. Matteo Richiardi, 2003. "The Promises and Perils of Agent-Based Computational Economics," LABORatorio R. Revelli Working Papers Series 29, LABORatorio R. Revelli, Centre for Employment Studies.
  7. Gaury, E.G.A. & Kleijnen, J.P.C. & Pierreval, H., 1998. "Customized Pull Systems for Single-Product Flow Lines," Discussion Paper 1998-117, Tilburg University, Center for Economic Research.
  8. Kleijnen, J.P.C., 2006. "Generalized Response Surface Methodology: A New Metaheuristic," Discussion Paper 2006-77, Tilburg University, Center for Economic Research.
  9. Kleijnen, J.P.C., 2001. "Experimental designs for sensitivity analysis of simulation models," Open Access publications from Tilburg University urn:nbn:nl:ui:12-86474, Tilburg University.
  10. Gaury, E.G.A. & Kleijnen, J.P.C. & Pierreval, H., 1997. "Configuring a pull production control strategy through a generic model," Discussion Paper 1997-101, Tilburg University, Center for Economic Research.
  11. Grazia Vicario & Daniele Romano, 2001. "Factorial experiments for sequential process," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(1-2), pages 191-204.


This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.


Access and download statistics


When requesting a correction, please mention this item's handle: RePEc:dgr:kubcen:199752. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Richard Broekman).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.