IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v51y2003i5p814-825.html
   My bibliography  Save this article

Using Ranking and Selection to “Clean Up” after Simulation Optimization

Author

Listed:
  • Justin Boesel

    (The MITRE Corporation, 1820 Dolley Madison Boulevard, McLean, Virginia 22102)

  • Barry L. Nelson

    (Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208-3119)

  • Seong-Hee Kim

    (School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332)

Abstract

In this paper we address the problem of finding the simulated system with the best (maximum or minimum) expected performance when the number of systems is large and initial samples from each system have already been taken. This problem may be encountered when a heuristic search procedure---perhaps one originally designed for use in a deterministic environment---has been applied in a simulation-optimization context. Because of stochastic variation, the system with the best sample mean at the end of the search procedure may not coincide with the true best system encountered during the search. This paper develops statistical procedures that return the best system encountered by the search (or one near the best) with a prespecified probability. We approach this problem using combinations of statistical subset selection and indifference-zone ranking procedures. The subset-selection procedures, which use only the data already collected, screen out the obviously inferior systems, while the indifference-zone procedures, which require additional simulation effort, distinguish the best from the less obviously inferior systems.

Suggested Citation

  • Justin Boesel & Barry L. Nelson & Seong-Hee Kim, 2003. "Using Ranking and Selection to “Clean Up” after Simulation Optimization," Operations Research, INFORMS, vol. 51(5), pages 814-825, October.
    • Handle: RePEc:inm:oropre:v:51:y:2003:i:5:p:814-825
    DOI: 10.1287/opre.51.5.814.16751
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/opre.51.5.814.16751
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.51.5.814.16751?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sidney Yakowitz & Pierre L'Ecuyer & Felisa Vázquez-Abad, 2000. "Global Stochastic Optimization with Low-Dispersion Point Sets," Operations Research, INFORMS, vol. 48(6), pages 939-950, December.
    2. Michael C. Fu, 2002. "Feature Article: Optimization for simulation: Theory vs. Practice," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 192-215, August.
    3. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    4. Barry L. Nelson & David Goldsman, 2001. "Comparisons with a Standard in Simulation Experiments," Management Science, INFORMS, vol. 47(3), pages 449-463, March.
    5. Barry L. Nelson & Frank J. Matejcik, 1995. "Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in Simulation," Management Science, INFORMS, vol. 41(12), pages 1935-1945, December.
    6. Stephen E. Chick & Koichiro Inoue, 2001. "New Two-Stage and Sequential Procedures for Selecting the Best Simulated System," Operations Research, INFORMS, vol. 49(5), pages 732-743, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jie Xu & Barry L. Nelson & L. Jeff Hong, 2013. "An Adaptive Hyperbox Algorithm for High-Dimensional Discrete Optimization via Simulation Problems," INFORMS Journal on Computing, INFORMS, vol. 25(1), pages 133-146, February.
    2. Kabirian, Alireza & Ólafsson, Sigurdur, 2011. "Continuous optimization via simulation using Golden Region search," European Journal of Operational Research, Elsevier, vol. 208(1), pages 19-27, January.
    3. Eric C. Ni & Dragos F. Ciocan & Shane G. Henderson & Susan R. Hunter, 2017. "Efficient Ranking and Selection in Parallel Computing Environments," Operations Research, INFORMS, vol. 65(3), pages 821-836, June.
    4. Zhongshun Shi & Siyang Gao & Hui Xiao & Weiwei Chen, 2019. "A worst‐case formulation for constrained ranking and selection with input uncertainty," Naval Research Logistics (NRL), John Wiley & Sons, vol. 66(8), pages 648-662, December.
    5. Miguel Lejeune & François Margot, 2011. "Optimization for simulation: LAD accelerator," Annals of Operations Research, Springer, vol. 188(1), pages 285-305, August.
    6. Batur, D. & Choobineh, F., 2010. "A quantile-based approach to system selection," European Journal of Operational Research, Elsevier, vol. 202(3), pages 764-772, May.
    7. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    8. Stephen E. Chick & Noah Gans, 2009. "Economic Analysis of Simulation Selection Problems," Management Science, INFORMS, vol. 55(3), pages 421-437, March.
    9. Sigrún Andradóttir & Andrei A. Prudius, 2009. "Balanced Explorative and Exploitative Search with Estimation for Simulation Optimization," INFORMS Journal on Computing, INFORMS, vol. 21(2), pages 193-208, May.
    10. Stephen E. Chick & Koichiro Inoue, 2001. "New Procedures to Select the Best Simulated System Using Common Random Numbers," Management Science, INFORMS, vol. 47(8), pages 1133-1149, August.
    11. Kleijnen, J.P.C. & Sanchez, S.M. & Lucas, T.W. & Cioppa, T.M., 2003. "A User's Guide to the Brave New World of Designing Simulation Experiments," Discussion Paper 2003-1, Tilburg University, Center for Economic Research.
    12. Satyajith Amaran & Nikolaos V. Sahinidis & Bikram Sharda & Scott J. Bury, 2016. "Simulation optimization: a review of algorithms and applications," Annals of Operations Research, Springer, vol. 240(1), pages 351-380, May.
    13. Sigrún Andradóttir, 2002. "Simulation Optimization: Integrating Research and Practice," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 216-219, August.
    14. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    15. Tsai, Shing Chih, 2011. "Selecting the best simulated system with weighted control-variate estimators," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 82(4), pages 705-717.
    16. Shing Chih Tsai & Tse Yang, 2017. "Rapid screening algorithms for stochastically constrained problems," Annals of Operations Research, Springer, vol. 254(1), pages 425-447, July.
    17. Tsai, Shing Chih & Chu, I-Hao, 2012. "Controlled multistage selection procedures for comparison with a standard," European Journal of Operational Research, Elsevier, vol. 223(3), pages 709-721.
    18. Chen, E. Jack & Kelton, W. David, 2005. "Sequential selection procedures: Using sample means to improve efficiency," European Journal of Operational Research, Elsevier, vol. 166(1), pages 133-153, October.
    19. Michael C. Fu, 2002. "Simulation Optimization in the Future: Evolution or Revolution?," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 226-227, August.
    20. Stephen E. Chick & Yaozhong Wu, 2005. "Selection Procedures with Frequentist Expected Opportunity Cost Bounds," Operations Research, INFORMS, vol. 53(5), pages 867-878, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:51:y:2003:i:5:p:814-825. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.