IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v47y2001i3p449-463.html
   My bibliography  Save this article

Comparisons with a Standard in Simulation Experiments

Author

Listed:
  • Barry L. Nelson

    (Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208)

  • David Goldsman

    (School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332)

Abstract

We consider the problem of comparing a finite number of stochastic systems with respect to a single system (designated as the "standard") via simulation experiments. The comparison is based on expected performance, and the goal is to determine if any system has larger expected performance than the standard, and if so to identify the best of the alternatives. In this paper we provide two-stage experiment design and analysis procedures to solve the problem for a variety of scenarios, including those in which we encounter unequal variances across systems, as well as those in which we use the variance reduction technique of common random numbers and it is appropriate to do so. The emphasis is added because in some cases common random numbers can be counterproductive when performing comparisons with a standard. We also provide methods for estimating the critical constants required by our procedures, present a portion of an extensive empirical study, and demonstrate one of the procedures via a numerical example.

Suggested Citation

  • Barry L. Nelson & David Goldsman, 2001. "Comparisons with a Standard in Simulation Experiments," Management Science, INFORMS, vol. 47(3), pages 449-463, March.
    • Handle: RePEc:inm:ormnsc:v:47:y:2001:i:3:p:449-463
    DOI: 10.1287/mnsc.47.3.449.9778
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.47.3.449.9778
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.47.3.449.9778?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Frank J. Matejcik & Barry L. Nelson, 1995. "Two-Stage Multiple Comparisons with the Best for Computer Simulation," Operations Research, INFORMS, vol. 43(4), pages 633-640, August.
    2. Wei-Ning Yang & Barry L. Nelson, 1991. "Using Common Random Numbers and Control Variates in Multiple-Comparison Procedures," Operations Research, INFORMS, vol. 39(4), pages 583-591, August.
    3. Paul Desruelle & Harold J. Steudel, 1996. "A Queuing Network Model of a Single-Operator Manufacturing Workcell with Machine/Operator Interference," Management Science, INFORMS, vol. 42(4), pages 576-590, April.
    4. Barry L. Nelson & Frank J. Matejcik, 1995. "Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in Simulation," Management Science, INFORMS, vol. 41(12), pages 1935-1945, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kleijnen, J.P.C. & Sanchez, S.M. & Lucas, T.W. & Cioppa, T.M., 2003. "A User's Guide to the Brave New World of Designing Simulation Experiments," Other publications TiSEM a6910d11-f9bc-4246-b1a7-2, Tilburg University, School of Economics and Management.
    2. Kabirian, Alireza & Ólafsson, Sigurdur, 2011. "Continuous optimization via simulation using Golden Region search," European Journal of Operational Research, Elsevier, vol. 208(1), pages 19-27, January.
    3. Tsai, Shing Chih & Chu, I-Hao, 2012. "Controlled multistage selection procedures for comparison with a standard," European Journal of Operational Research, Elsevier, vol. 223(3), pages 709-721.
    4. Batur, D. & Choobineh, F., 2010. "A quantile-based approach to system selection," European Journal of Operational Research, Elsevier, vol. 202(3), pages 764-772, May.
    5. Jing Xie & Peter I. Frazier, 2013. "Sequential Bayes-Optimal Policies for Multiple Comparisons with a Known Standard," Operations Research, INFORMS, vol. 61(5), pages 1174-1189, October.
    6. Demet Batur & Lina Wang & F. Fred Choobineh, 2018. "Methods for System Selection Based on Sequential Mean–Variance Analysis," INFORMS Journal on Computing, INFORMS, vol. 30(4), pages 724-738, November.
    7. Justin Boesel & Barry L. Nelson & Seong-Hee Kim, 2003. "Using Ranking and Selection to “Clean Up” after Simulation Optimization," Operations Research, INFORMS, vol. 51(5), pages 814-825, October.
    8. L. Jeff Hong & Jun Luo & Barry L. Nelson, 2015. "Chance Constrained Selection of the Best," INFORMS Journal on Computing, INFORMS, vol. 27(2), pages 317-334, May.
    9. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    10. Pichitlamken, Juta & Nelson, Barry L. & Hong, L. Jeff, 2006. "A sequential procedure for neighborhood selection-of-the-best in optimization via simulation," European Journal of Operational Research, Elsevier, vol. 173(1), pages 283-298, August.
    11. Zhongshun Shi & Yijie Peng & Leyuan Shi & Chun-Hung Chen & Michael C. Fu, 2022. "Dynamic Sampling Allocation Under Finite Simulation Budget for Feasibility Determination," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 557-568, January.
    12. Stephen E. Chick & Noah Gans, 2009. "Economic Analysis of Simulation Selection Problems," Management Science, INFORMS, vol. 55(3), pages 421-437, March.
    13. Satyajith Amaran & Nikolaos V. Sahinidis & Bikram Sharda & Scott J. Bury, 2016. "Simulation optimization: a review of algorithms and applications," Annals of Operations Research, Springer, vol. 240(1), pages 351-380, May.
    14. Hong Wan & Bruce E. Ankenman & Barry L. Nelson, 2006. "Controlled Sequential Bifurcation: A New Factor-Screening Method for Discrete-Event Simulation," Operations Research, INFORMS, vol. 54(4), pages 743-755, August.
    15. Shing Chih Tsai & Tse Yang, 2017. "Rapid screening algorithms for stochastically constrained problems," Annals of Operations Research, Springer, vol. 254(1), pages 425-447, July.
    16. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    17. Jürgen Branke & Stephen E. Chick & Christian Schmidt, 2007. "Selecting a Selection Procedure," Management Science, INFORMS, vol. 53(12), pages 1916-1932, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael C. Fu & Jian-Qiang Hu & Chun-Hung Chen & Xiaoping Xiong, 2007. "Simulation Allocation for Determining the Best Design in the Presence of Correlated Sampling," INFORMS Journal on Computing, INFORMS, vol. 19(1), pages 101-111, February.
    2. Diana M. Negoescu & Peter I. Frazier & Warren B. Powell, 2011. "The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery," INFORMS Journal on Computing, INFORMS, vol. 23(3), pages 346-363, August.
    3. Huashuai Qu & Ilya O. Ryzhov & Michael C. Fu & Zi Ding, 2015. "Sequential Selection with Unknown Correlation Structures," Operations Research, INFORMS, vol. 63(4), pages 931-948, August.
    4. Stephen E. Chick & Koichiro Inoue, 2001. "New Two-Stage and Sequential Procedures for Selecting the Best Simulated System," Operations Research, INFORMS, vol. 49(5), pages 732-743, October.
    5. Jing Xie & Peter I. Frazier & Stephen E. Chick, 2016. "Bayesian Optimization via Simulation with Pairwise Sampling and Correlated Prior Beliefs," Operations Research, INFORMS, vol. 64(2), pages 542-559, April.
    6. Nakayama, Marvin K., 2007. "Fixed-width multiple-comparison procedures using common random numbers for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 182(3), pages 1330-1349, November.
    7. Oliver Hinz & Jochen Eckert, 2010. "The Impact of Search and Recommendation Systems on Sales in Electronic Commerce," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 2(2), pages 67-77, April.
    8. Haque, Lani & Armstrong, Michael J., 2007. "A survey of the machine interference problem," European Journal of Operational Research, Elsevier, vol. 179(2), pages 469-482, June.
    9. Francis, Peter & Zhang, Guangming & Smilowitz, Karen, 2007. "Improved modeling and solution methods for the multi-resource routing problem," European Journal of Operational Research, Elsevier, vol. 180(3), pages 1045-1059, August.
    10. Wang, Honggang, 2012. "Retrospective optimization of mixed-integer stochastic systems using dynamic simplex linear interpolation," European Journal of Operational Research, Elsevier, vol. 217(1), pages 141-148.
    11. Stephen E. Chick & Koichiro Inoue, 2001. "New Procedures to Select the Best Simulated System Using Common Random Numbers," Management Science, INFORMS, vol. 47(8), pages 1133-1149, August.
    12. Anton J. Kleywegt & Vijay S. Nori & Martin W. P. Savelsbergh, 2002. "The Stochastic Inventory Routing Problem with Direct Deliveries," Transportation Science, INFORMS, vol. 36(1), pages 94-118, February.
    13. Sigurdur Ólafsson, 2004. "Two-Stage Nested Partitions Method for Stochastic Optimization," Methodology and Computing in Applied Probability, Springer, vol. 6(1), pages 5-27, March.
    14. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    15. Tsai, Shing Chih, 2011. "Selecting the best simulated system with weighted control-variate estimators," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 82(4), pages 705-717.
    16. Jacobson, Sheldon H. & McLay, Laura A., 2009. "Applying statistical tests to empirically compare tabu search parameters for MAX 3-SATISFIABILITY: A case study," Omega, Elsevier, vol. 37(3), pages 522-534, June.
    17. Wallace J. Hopp & Seyed M.R. Iravani & Biying Shou & Robert Lien, 2009. "Design and control of agile automated CONWIP production lines," Naval Research Logistics (NRL), John Wiley & Sons, vol. 56(1), pages 42-56, February.
    18. Rafiei, Rezvan & Nourelfath, Mustapha & Gaudreault, Jonathan & De Santa-Eulalia, Luis Antonio & Bouchard, Mathieu, 2015. "Dynamic safety stock in co-production demand-driven wood remanufacturing mills: A case study," International Journal of Production Economics, Elsevier, vol. 165(C), pages 90-99.
    19. Eric C. Ni & Dragos F. Ciocan & Shane G. Henderson & Susan R. Hunter, 2017. "Efficient Ranking and Selection in Parallel Computing Environments," Operations Research, INFORMS, vol. 65(3), pages 821-836, June.
    20. Anton J. Kleywegt & Vijay S. Nori & Martin W. P. Savelsbergh, 2004. "Dynamic Programming Approximations for a Stochastic Inventory Routing Problem," Transportation Science, INFORMS, vol. 38(1), pages 42-70, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:47:y:2001:i:3:p:449-463. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.