IDEAS home Printed from https://ideas.repec.org/a/inm/orijoc/v35y2023i2p350-367.html
   My bibliography  Save this article

Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms

Author

Listed:
  • David J. Eckman

    (Wm Michael Barnes ’64 Department of Industrial and Systems Engineering, Texas A&M University, College Station, Texas 77843)

  • Shane G. Henderson

    (School of Operations Research and Information Engineering, Cornell University, Ithaca, New York 14853)

  • Sara Shashaani

    (Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, North Carolina 27695)

Abstract

Simulation optimization involves optimizing some objective function that can only be estimated via stochastic simulation. Many important problems can be profitably viewed within this framework. Whereas many solvers—implementations of simulation-optimization algorithms—exist or are in development, comparisons among solvers are not standardized and are often limited in scope. Such comparisons help advance solver development, clarify the relative performance of solvers, and identify classes of problems that defy efficient solution, among many other uses. We develop performance measures and plots, and estimators thereof, to evaluate and compare solvers and diagnose their strengths and weaknesses on a testbed of simulation-optimization problems. We explain the need for two-level simulation in this context and provide supporting convergence theory. We also describe how to use bootstrapping to obtain error estimates for the estimators.

Suggested Citation

  • David J. Eckman & Shane G. Henderson & Sara Shashaani, 2023. "Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms," INFORMS Journal on Computing, INFORMS, vol. 35(2), pages 350-367, March.
    • Handle: RePEc:inm:orijoc:v:35:y:2023:i:2:p:350-367
    DOI: 10.1287/ijoc.2022.1261
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/ijoc.2022.1261
    Download Restriction: no
    File URL: https://libkey.io/10.1287/ijoc.2022.1261?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    2. Michael C. Fu, 2002. "Feature Article: Optimization for simulation: Theory vs. Practice," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 192-215, August.
    3. Kuo-Hao Chang & L. Jeff Hong & Hong Wan, 2013. "Stochastic Trust-Region Response-Surface Method (STRONG)---A New Response-Surface Framework for Simulation Optimization," INFORMS Journal on Computing, INFORMS, vol. 25(2), pages 230-243, May.
    4. Mark Broadie & Yiping Du & Ciamac C. Moallemi, 2015. "Risk Estimation via Regression," Operations Research, INFORMS, vol. 63(5), pages 1077-1097, October.
    5. Michael B. Gordy & Sandeep Juneja, 2010. "Nested Simulation in Portfolio Risk Measurement," Management Science, INFORMS, vol. 56(10), pages 1833-1848, October.
    6. Kyle Cooper & Susan R. Hunter & Kalyani Nagaraj, 2020. "Biobjective Simulation Optimization on Integer Lattices Using the Epsilon-Constraint Method in a Retrospective Approximation Framework," INFORMS Journal on Computing, INFORMS, vol. 32(4), pages 1080-1100, October.
    7. Peter W. Glynn, 2002. "Additional Perspectives on Simulation for Optimization," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 220-222, August.
    8. Mark Broadie & Yiping Du & Ciamac C. Moallemi, 2011. "Efficient Risk Estimation via Nested Sequential Simulation," Management Science, INFORMS, vol. 57(6), pages 1172-1194, June.
    9. Nicholas Gould & Dominique Orban & Philippe Toint, 2015. "CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization," Computational Optimization and Applications, Springer, vol. 60(3), pages 545-557, April.
    10. Satyajith Amaran & Nikolaos V. Sahinidis & Bikram Sharda & Scott J. Bury, 2016. "Simulation optimization: a review of algorithms and applications," Annals of Operations Research, Springer, vol. 240(1), pages 351-380, May.
    11. Russell R. Barton & John S. Ivey, Jr., 1996. "Nelder-Mead Simplex Modifications for Simulation Optimization," Management Science, INFORMS, vol. 42(7), pages 954-973, July.
    12. Marie Chau & Michael C. Fu, 2015. "An Overview of Stochastic Approximation," International Series in Operations Research & Management Science, in: Michael C Fu (ed.), Handbook of Simulation Optimization, edition 127, chapter 0, pages 149-178, Springer.
    13. Yunpeng Sun & Daniel W. Apley & Jeremy Staum, 2011. "Efficient Nested Simulation for Estimating the Variance of a Conditional Expectation," Operations Research, INFORMS, vol. 59(4), pages 998-1007, August.
    14. Saeed Ghadimi & Guanghui Lan, 2015. "Stochastic Approximation Methods and Their Finite-Time Convergence Properties," International Series in Operations Research & Management Science, in: Michael C Fu (ed.), Handbook of Simulation Optimization, edition 127, chapter 0, pages 179-206, Springer.
    15. Michael C. Fu, 2015. "Overview of the Handbook," International Series in Operations Research & Management Science, in: Michael C Fu (ed.), Handbook of Simulation Optimization, edition 127, chapter 0, pages 1-7, Springer.
    16. L. Jeff Hong & Sandeep Juneja & Guangwu Liu, 2017. "Kernel Smoothing for Nested Estimation with Application to Portfolio Risk Measurement," Operations Research, INFORMS, vol. 65(3), pages 657-673, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David J. Eckman & Shane G. Henderson & Sara Shashaani, 2023. "SimOpt: A Testbed for Simulation-Optimization Experiments," INFORMS Journal on Computing, INFORMS, vol. 35(2), pages 495-508, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mingbin Ben Feng & Eunhye Song, 2020. "Optimal Nested Simulation Experiment Design via Likelihood Ratio Method," Papers 2008.13087, arXiv.org, revised Jul 2021.
    2. Liu, Xiaoyu & Yan, Xing & Zhang, Kun, 2024. "Kernel quantile estimators for nested simulation with application to portfolio value-at-risk measurement," European Journal of Operational Research, Elsevier, vol. 312(3), pages 1168-1177.
    3. Guangxin Jiang & L. Jeff Hong & Barry L. Nelson, 2020. "Online Risk Monitoring Using Offline Simulation," INFORMS Journal on Computing, INFORMS, vol. 32(2), pages 356-375, April.
    4. Wang, Tianxiang & Xu, Jie & Hu, Jian-Qiang & Chen, Chun-Hung, 2023. "Efficient estimation of a risk measure requiring two-stage simulation optimization," European Journal of Operational Research, Elsevier, vol. 305(3), pages 1355-1365.
    5. Runhuan Feng & Peng Li, 2021. "Sample Recycling Method -- A New Approach to Efficient Nested Monte Carlo Simulations," Papers 2106.06028, arXiv.org.
    6. Kun Zhang & Ben Mingbin Feng & Guangwu Liu & Shiyu Wang, 2022. "Sample Recycling for Nested Simulation with Application in Portfolio Risk Measurement," Papers 2203.15929, arXiv.org.
    7. Dang, Ou & Feng, Mingbin & Hardy, Mary R., 2023. "Two-stage nested simulation of tail risk measurement: A likelihood ratio approach," Insurance: Mathematics and Economics, Elsevier, vol. 108(C), pages 1-24.
    8. Qiyun Pan & Eunshin Byon & Young Myoung Ko & Henry Lam, 2020. "Adaptive importance sampling for extreme quantile estimation with stochastic black box computer models," Naval Research Logistics (NRL), John Wiley & Sons, vol. 67(7), pages 524-547, October.
    9. Wen Shi & Xi Chen, 2018. "Efficient budget allocation strategies for elementary effects method in stochastic simulation," Naval Research Logistics (NRL), John Wiley & Sons, vol. 65(3), pages 218-241, April.
    10. Feng, Ben Mingbin & Li, Johnny Siu-Hang & Zhou, Kenneth Q., 2022. "Green nested simulation via likelihood ratio: Applications to longevity risk management," Insurance: Mathematics and Economics, Elsevier, vol. 106(C), pages 285-301.
    11. Alessandro Gnoatto & Athena Picarelli & Christoph Reisinger, 2020. "Deep xVA solver -- A neural network based counterparty credit risk management framework," Papers 2005.02633, arXiv.org, revised Dec 2022.
    12. Mark Broadie & Yiping Du & Ciamac C. Moallemi, 2015. "Risk Estimation via Regression," Operations Research, INFORMS, vol. 63(5), pages 1077-1097, October.
    13. Satyajith Amaran & Nikolaos V. Sahinidis & Bikram Sharda & Scott J. Bury, 2016. "Simulation optimization: a review of algorithms and applications," Annals of Operations Research, Springer, vol. 240(1), pages 351-380, May.
    14. Patrick Cheridito & John Ery & Mario V. Wüthrich, 2020. "Assessing Asset-Liability Risk with Neural Networks," Risks, MDPI, vol. 8(1), pages 1-17, February.
    15. Cornelis S. L. de Graaf & Drona Kandhai & Christoph Reisinger, 2016. "Efficient exposure computation by risk factor decomposition," Papers 1608.01197, arXiv.org, revised Feb 2018.
    16. Lucio Fernandez‐Arjona & Damir Filipović, 2022. "A machine learning approach to portfolio pricing and risk management for high‐dimensional problems," Mathematical Finance, Wiley Blackwell, vol. 32(4), pages 982-1019, October.
    17. Helin Zhu & Tianyi Liu & Enlu Zhou, 2015. "Risk Quantification in Stochastic Simulation under Input Uncertainty," Papers 1507.06015, arXiv.org, revised Dec 2017.
    18. Hongjun Ha & Daniel Bauer, 2022. "A least-squares Monte Carlo approach to the estimation of enterprise risk," Finance and Stochastics, Springer, vol. 26(3), pages 417-459, July.
    19. Patrick Cheridito & John Ery & Mario V. Wuthrich, 2021. "Assessing asset-liability risk with neural networks," Papers 2105.12432, arXiv.org.
    20. L. Jeff Hong & Sandeep Juneja & Guangwu Liu, 2017. "Kernel Smoothing for Nested Estimation with Application to Portfolio Risk Measurement," Operations Research, INFORMS, vol. 65(3), pages 657-673, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orijoc:v:35:y:2023:i:2:p:350-367. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.