IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v47y2001i11p1533-1538.html
   My bibliography  Save this article

Validation of Trace-Driven Simulation Models: Bootstrap Tests

Author

Listed:
  • Jack P. C. Kleijnen

    (Department of Information Systems (BIK)/Center for Economic Research (CentER), Tilburg University (KUB), 5000 LE Tilburg, Netherlands)

  • Russell C. H. Cheng

    (Department of Mathematical Sciences, University of Southampton, Highfield, Southampton, SO17 1BJ, United Kingdom)

  • Bert Bettonvil

    (Department of Information Systems (BIK)/Center for Economic Research (CentER), Tilburg University (KUB), 5000 LE Tilburg, Netherlands)

Abstract

Trace-driven (or correlated inspection) simulation means that the simulated and the real systems have some common inputs (say, historical arrival times) so that the two systems' outputs are cross-correlated. To validate such a simulation, this paper focuses on the difference between the average simulated and real responses. To evaluate this validation statistic, the paper develops a novel bootstrap technique---based on replicated runs. This validation statistic and the bootstrap technique are evaluated in extensive Monte Carlo experiments with specific single-server queues. These experiments show acceptable Type-I and Type-II error probabilities.

Suggested Citation

  • Jack P. C. Kleijnen & Russell C. H. Cheng & Bert Bettonvil, 2001. "Validation of Trace-Driven Simulation Models: Bootstrap Tests," Management Science, INFORMS, vol. 47(11), pages 1533-1538, November.
    • Handle: RePEc:inm:ormnsc:v:47:y:2001:i:11:p:1533-1538
    DOI: 10.1287/mnsc.47.11.1533.10255
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.47.11.1533.10255
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.47.11.1533.10255?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
    2. Kleijnen, J.P.C., 2000. "Strategic directions in verification, validation and accreditation research : A personal view," Other publications TiSEM eee4931d-abef-4137-8112-7, Tilburg University, School of Economics and Management.
    3. Bruce Schmeiser, 1982. "Batch Size Effects in the Analysis of Simulation Output," Operations Research, INFORMS, vol. 30(3), pages 556-568, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Deflandre, D. & Kleijnen, J.P.C., 2002. "Statistical Analysis of Random Simulations : Bootstrap Tutorial," Other publications TiSEM 351056e6-c63d-4fec-8c00-a, Tilburg University, School of Economics and Management.
    2. Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Discussion Paper 2007-09, Tilburg University, Center for Economic Research.
    3. Kleijnen, Jack P.C., 2011. "Ethical issues in engineering models : An operations researcher's reflections," Other publications TiSEM 88caba9e-1d78-47d7-9a0b-b, Tilburg University, School of Economics and Management.
    4. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Enver Yücesan, 1993. "Randomization tests for initialization bias in simulation output," Naval Research Logistics (NRL), John Wiley & Sons, vol. 40(5), pages 643-663, August.
    2. Song, Wheyming Tina, 1996. "On the estimation of optimal batch sizes in the analysis of simulation output," European Journal of Operational Research, Elsevier, vol. 88(2), pages 304-319, January.
    3. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    4. David F. Muñoz & Peter W. Glynn, 2001. "Multivariate Standardized Time Series for Steady-State Simulation Output Analysis," Operations Research, INFORMS, vol. 49(3), pages 413-422, June.
    5. David Goldsman & Keebom Kang & Seong‐Hee Kim & Andrew F. Seila & Gamze Tokol, 2007. "Combining standardized time series area and Cramér–von Mises variance estimators," Naval Research Logistics (NRL), John Wiley & Sons, vol. 54(4), pages 384-396, June.
    6. Mingchang Chih, 2019. "An Insight into the Data Structure of the Dynamic Batch Means Algorithm with Binary Tree Code," Mathematics, MDPI, vol. 7(9), pages 1-8, August.
    7. van Schaik, F.D.J. & Kleijnen, J.P.C., 2001. "Sealed-Bid Auctions : Case Study," Other publications TiSEM dd93bf40-d790-4fd1-ac2c-f, Tilburg University, School of Economics and Management.
    8. Kleijnen, Jack P. C. & Sargent, Robert G., 2000. "A methodology for fitting and validating metamodels in simulation," European Journal of Operational Research, Elsevier, vol. 120(1), pages 14-29, January.
    9. Christos Alexopoulos & Nilay Tanık Argon & David Goldsman & Gamze Tokol & James R. Wilson, 2007. "Overlapping Variance Estimators for Simulation," Operations Research, INFORMS, vol. 55(6), pages 1090-1103, December.
    10. Kannan, V. R. & Ghosh, S., 1995. "Using dynamic cellular manufacturing to simplify scheduling in cell based production systems," Omega, Elsevier, vol. 23(4), pages 443-452, August.
    11. Christos Alexopoulos & David Goldsman & Gamze Tokol, 2001. "Properties of Batched Quadratic-Form Variance Parameter Estimators for Simulations," INFORMS Journal on Computing, INFORMS, vol. 13(2), pages 149-156, May.
    12. Rizzoli, Andrea E. & Fornara, Nicoletta & Gambardella, Luca Maria, 2002. "A simulation tool for combined rail/road transport in intermodal terminals," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 59(1), pages 57-71.
    13. Driessen, M.A. & van Houtum, G.J. & Zijm, W.H.M. & Rustenburg, W.D., 2020. "Capacity assignment in repair shops with high material uncertainty," International Journal of Production Economics, Elsevier, vol. 221(C).
    14. Song, Wheyming T. & Chih, Mingchang, 2010. "Extended dynamic partial-overlapping batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 203(3), pages 640-651, June.
    15. Kleijnen, J.P.C., 1999. "Validation of Models : Statistical Techniques and Data Availability," Discussion Paper 1999-104, Tilburg University, Center for Economic Research.
    16. Kleijnen, J.P.C., 2001. "Experimental Design for Sensitivity Analysis of Simulation Models," Other publications TiSEM 1c716145-1d34-41b5-b965-7, Tilburg University, School of Economics and Management.
    17. Srinagesh Gavirneni & Douglas J. Morrice & Peter Mullarkey, 2004. "Simulation Helps Maxager Shorten Its Sales Cycle," Interfaces, INFORMS, vol. 34(2), pages 87-96, April.
    18. Kenneth W. Bauer & James R. Wilson, 1992. "Control‐variate selection criteria," Naval Research Logistics (NRL), John Wiley & Sons, vol. 39(3), pages 307-321, April.
    19. Christos Alexopoulos & Nilay Tanık Argon & David Goldsman & Natalie M. Steiger & Gamze Tokol & James R. Wilson, 2007. "Efficient Computation of Overlapping Variance Estimators for Simulation," INFORMS Journal on Computing, INFORMS, vol. 19(3), pages 314-327, August.
    20. Legato, Pasquale & Mazza, Rina M., 2001. "Berth planning and resources optimisation at a container terminal via discrete event simulation," European Journal of Operational Research, Elsevier, vol. 133(3), pages 537-547, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:47:y:2001:i:11:p:1533-1538. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.