IDEAS home Printed from https://ideas.repec.org/p/tiu/tiucen/66449de2-3d9c-4d19-a3e4-ada1b703bcde.html
   My bibliography  Save this paper

Simulation Experiments in Practice : Statistical Design and Regression Analysis

Author

Listed:
  • Kleijnen, J.P.C.

    (Tilburg University, Center For Economic Research)

Abstract

In practice, simulation analysts often change only one factor at a time, and use graphical analysis of the resulting Input/Output (I/O) data. The goal of this article is to change these traditional, naïve methods of design and analysis, because statistical theory proves that more information is obtained when applying Design Of Experiments (DOE) and linear regression analysis. Unfortunately, classic DOE and regression analysis assume a single simulation response that is normally and independently distributed with a constant variance; moreover, the regression (meta)model of the simulation model’s I/O behaviour is assumed to have residuals with zero means. This article addresses the following practical questions: (i) How realistic are these assumptions, in practice? (ii) How can these assumptions be tested? (iii) If assumptions are violated, can the simulation's I/O data be transformed such that the assumptions do hold? (iv) If not, which alternative statistical methods can then be applied?
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(Th
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Discussion Paper 2007-09, Tilburg University, Center for Economic Research.
    • Handle: RePEc:tiu:tiucen:66449de2-3d9c-4d19-a3e4-ada1b703bcde
    as

    Download full text from publisher

    To our knowledge, this item is not available for download. To find whether it is available, there are three options:
    1. Check below whether another version of this item is available online.
    2. Check on the provider's web page whether it is in fact available.
    3. Perform a search for a similarly titled item that would be available.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jack P. C. Kleijnen & Russell C. H. Cheng & Bert Bettonvil, 2001. "Validation of Trace-Driven Simulation Models: Bootstrap Tests," Management Science, INFORMS, vol. 47(11), pages 1533-1538, November.
    2. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.
    3. Kleijnen, J.P.C. & van Groenendaal, W.J.H., 1992. "Two-stage versus sequential sample-size determination in regression analysis of simulation experiments," Other publications TiSEM cac61c9f-0967-4f5a-aab1-9, Tilburg University, School of Economics and Management.
    4. van Beers, W.C.M. & Kleijnen, J.P.C., 2005. "Customized Sequential Designs for Random Simulation Experiments : Kriging Metamodelling and Bootstrapping," Other publications TiSEM 893d38f9-8ca5-42ae-9737-6, Tilburg University, School of Economics and Management.
    5. Arcones, Miguel A. & Wang, Yishi, 2006. "Some new tests for normality based on U-processes," Statistics & Probability Letters, Elsevier, vol. 76(1), pages 69-82, January.
    6. J.P.C. Kleijnen & J. Kriens & H. Timmermans & H. van den Wildenberg, 1989. "Regression Sampling in Statistical Auditing: A Practical Survey and Evaluation," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 43(4), pages 193-209, December.
    7. van Beers, Wim C.M. & Kleijnen, Jack P.C., 2008. "Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping," European Journal of Operational Research, Elsevier, vol. 186(3), pages 1099-1113, May.
    8. Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
    9. Kleijnen, J.P.C. & Cheng, R.C.H. & Bettonvil, B.W.M., 2001. "Validation of trace-driven simulation models : Bootstrapped tests," Other publications TiSEM 13d22f9a-c1e2-45a7-b96a-9, Tilburg University, School of Economics and Management.
    10. Davidson, Russell & MacKinnon, James G., 2007. "Improving the reliability of bootstrap tests with the fast double bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 51(7), pages 3259-3281, April.
    11. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Other publications TiSEM d8c37ad3-f9a5-4824-986d-2, Tilburg University, School of Economics and Management.
    12. Kleijnen, J.P.C. & van Beers, W.C.M., 2003. "Application-driven Sequential Designs for Simulation Experiments : Kriging Metamodeling," Other publications TiSEM 1af5ac49-a5f3-4b8a-901c-b, Tilburg University, School of Economics and Management.
    13. Ayanso, Anteneh & Diaby, Moustapha & Nair, Suresh K., 2006. "Inventory rationing via drop-shipping in Internet retailing: A sensitivity analysis," European Journal of Operational Research, Elsevier, vol. 171(1), pages 135-152, May.
    14. J P C Kleijnen & W C M van Beers, 2004. "Application-driven sequential designs for simulation experiments: Kriging metamodelling," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 55(8), pages 876-883, August.
    15. Ruud, Paul A., 2000. "An Introduction to Classical Econometric Theory," OUP Catalogue, Oxford University Press, number 9780195111644, Decembrie.
    16. V C Ivănescu & J W M Bertrand & J C Fransoo & J P C Kleijnen, 2006. "Bootstrapping to solve the limited data problem in production control: an application in batch process industries," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 57(1), pages 2-9, January.
    17. Freeman, Jade & Modarres, Reza, 2006. "Inverse Box-Cox: The power-normal distribution," Statistics & Probability Letters, Elsevier, vol. 76(8), pages 764-772, April.
    18. Matias Salibian-Barrera, 2006. "The Asymptotics of MM-Estimators for Linear Regression with Fixed Designs," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 63(3), pages 283-294, June.
    19. Godfrey, L.G., 2006. "Tests for regression models with heteroskedasticity of unknown form," Computational Statistics & Data Analysis, Elsevier, vol. 50(10), pages 2715-2733, June.
    20. Helton, J.C. & Johnson, J.D. & Sallaberry, C.J. & Storlie, C.B., 2006. "Survey of sampling-based methods for uncertainty and sensitivity analysis," Reliability Engineering and System Safety, Elsevier, vol. 91(10), pages 1175-1209.
    21. Salibian-Barrera, Matias, 2006. "Bootstrapping MM-estimators for linear regression with fixed designs," Statistics & Probability Letters, Elsevier, vol. 76(12), pages 1287-1297, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Other publications TiSEM d8c37ad3-f9a5-4824-986d-2, Tilburg University, School of Economics and Management.
    2. Scott L. Rosen & Christopher P. Saunders & Samar K Guharay, 2015. "A Structured Approach for Rapidly Mapping Multilevel System Measures via Simulation Metamodeling," Systems Engineering, John Wiley & Sons, vol. 18(1), pages 87-101, January.
    3. Kleijnen, J.P.C. & van Beers, W.C.M. & van Nieuwenhuyse, I., 2008. "Constrained Optimization in Simulation : A Novel Approach," Other publications TiSEM e49ba0fc-853c-4a13-b564-d, Tilburg University, School of Economics and Management.
    4. Kleijnen, J.P.C. & van Beers, W.C.M. & van Nieuwenhuyse, I., 2008. "Constrained Optimization in Simulation : A Novel Approach," Other publications TiSEM e49ba0fc-853c-4a13-b564-d, Tilburg University, School of Economics and Management.
    5. Kleijnen, Jack P.C., 2009. "Kriging metamodeling in simulation: A review," European Journal of Operational Research, Elsevier, vol. 192(3), pages 707-716, February.
    6. Song, Kunling & Zhang, Yugang & Shen, Linjie & Zhao, Qingyan & Song, Bifeng, 2021. "A failure boundary exploration and exploitation framework combining adaptive Kriging model and sample space partitioning strategy for efficient reliability analysis," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    7. Kleijnen, Jack P.C., 2017. "Regression and Kriging metamodels with their experimental designs in simulation: A review," European Journal of Operational Research, Elsevier, vol. 256(1), pages 1-16.
    8. Plischke, Elmar & Borgonovo, Emanuele & Smith, Curtis L., 2013. "Global sensitivity measures from given data," European Journal of Operational Research, Elsevier, vol. 226(3), pages 536-550.
    9. Kleijnen, Jack P.C. & Beers, Wim van & Nieuwenhuyse, Inneke van, 2010. "Constrained optimization in expensive simulation: Novel approach," European Journal of Operational Research, Elsevier, vol. 202(1), pages 164-174, April.
    10. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.
    11. M I Reis dos Santos & P M Reis dos Santos, 2011. "Construction and validation of distribution-based regression simulation metamodels," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(7), pages 1376-1384, July.
    12. Kleijnen, J.P.C., 2015. "Regression and Kriging Metamodels with Their Experimental Designs in Simulation : Review," Other publications TiSEM c592e895-1656-43c3-8c7e-f, Tilburg University, School of Economics and Management.
    13. Gan, Guojun & Lin, X. Sheldon, 2015. "Valuation of large variable annuity portfolios under nested simulation: A functional data approach," Insurance: Mathematics and Economics, Elsevier, vol. 62(C), pages 138-150.
    14. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    15. Edwin Dam & Bart Husslage & Dick Hertog, 2010. "One-dimensional nested maximin designs," Journal of Global Optimization, Springer, vol. 46(2), pages 287-306, February.
    16. Maroussa Zagoraiou & Alessandro Baldi Antognini, 2009. "Optimal designs for parameter estimation of the Ornstein–Uhlenbeck process," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 25(5), pages 583-600, September.
    17. Mlakar, Miha & Petelin, Dejan & Tušar, Tea & Filipič, Bogdan, 2015. "GP-DEMO: Differential Evolution for Multiobjective Optimization based on Gaussian Process models," European Journal of Operational Research, Elsevier, vol. 243(2), pages 347-361.
    18. Arreola-Risa, Antonio & Giménez-García, Víctor M. & Martínez-Parra, José Luis, 2011. "Optimizing stochastic production-inventory systems: A heuristic based on simulation and regression analysis," European Journal of Operational Research, Elsevier, vol. 213(1), pages 107-118, August.
    19. Bruce Ankenman & Barry L. Nelson & Jeremy Staum, 2010. "Stochastic Kriging for Simulation Metamodeling," Operations Research, INFORMS, vol. 58(2), pages 371-382, April.
    20. Cheng, R.C.H. & Kleijnen, J.P.C., 1996. "Improved design of queuing simulation experiments with highly heteroscedastic responses," Other publications TiSEM d8a325a9-82bf-4d23-839e-2, Tilburg University, School of Economics and Management.

    More about this item

    Keywords

    metamodels; experimental designs; generalized least squares; multivariate analysis; normality; jackknife; bootstrap; heteroscedasticity; common random numbers; validation;
    All these keywords.

    JEL classification:

    • C0 - Mathematical and Quantitative Methods - - General
    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tiu:tiucen:66449de2-3d9c-4d19-a3e4-ada1b703bcde. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Richard Broekman (email available below). General contact details of provider: http://center.uvt.nl .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.