IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v67y2019i5p1362-1382.html
   My bibliography  Save this article

Optimization-Based Calibration of Simulation Input Models

Author

Listed:
  • Aleksandrina Goeva

    (Broad Institute, Cambridge, Massachusetts 02142)

  • Henry Lam

    (Department of Industrial Engineering and Operations Research, Columbia University, New York, New York 10027)

  • Huajie Qian

    (Department of Industrial Engineering and Operations Research, Columbia University, New York, New York 10027)

  • Bo Zhang

    (IBM Research AI, Yorktown Heights, New York 10598)

Abstract

Studies on simulation input uncertainty are often built on the availability of input data. In this paper, we investigate an inverse problem where, given only the availability of output data, we nonparametrically calibrate the input models and other related performance measures of interest. We propose an optimization-based framework to compute statistically valid bounds on input quantities. The framework utilizes constraints that connect the statistical information of the real-world outputs with the input–output relation via a simulable map. We analyze the statistical guarantees of this approach from the view of data-driven distributionally robust optimization, and show how they relate to the function complexity of the constraints arising in our framework. We investigate an iterative procedure based on a stochastic quadratic penalty method to approximately solve the resulting optimization. We conduct numerical experiments to demonstrate our performances in bounding the input models and related quantities.

Suggested Citation

  • Aleksandrina Goeva & Henry Lam & Huajie Qian & Bo Zhang, 2019. "Optimization-Based Calibration of Simulation Input Models," Operations Research, INFORMS, vol. 67(5), pages 1362-1382, September.
    • Handle: RePEc:inm:oropre:v:67:y:2019:i:5:p:1362-1382
    DOI: opre.2018.1801
    as

    Download full text from publisher

    File URL: https://doi.org/opre.2018.1801
    Download Restriction: no
    File URL: https://libkey.io/opre.2018.1801?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ioana Popescu, 2005. "A Semidefinite Programming Approach to Optimal-Moment Bounds for Convex Classes of Distributions," Mathematics of Operations Research, INFORMS, vol. 30(3), pages 632-657, August.
    2. Aharon Ben-Tal & Dick den Hertog & Anja De Waegenaere & Bertrand Melenberg & Gijs Rennen, 2013. "Robust Solutions of Optimization Problems Affected by Uncertain Probabilities," Management Science, INFORMS, vol. 59(2), pages 341-357, April.
    3. Paul Glasserman & Xingbo Xu, 2013. "Robust Portfolio Control with Stochastic Factor Dynamics," Operations Research, INFORMS, vol. 61(4), pages 874-893, August.
    4. James E. Smith, 1995. "Generalized Chebychev Inequalities: Theory and Applications in Decision Analysis," Operations Research, INFORMS, vol. 43(5), pages 807-825, October.
    5. Tsung-Yin Wang & Jau-Chuan Ke & Kuo-Hsiung Wang & Siu-Chuen Ho, 2006. "Maximum Likelihood Estimates and Confidence Intervals of an M/M/R Queue with Heterogeneous Servers," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 63(2), pages 371-384, May.
    6. Basawa, I.V. & Bhat, U.N. & Zhou, J., 2008. "Parameter estimation using partial information with applications to queueing and related models," Statistics & Probability Letters, Elsevier, vol. 78(12), pages 1375-1383, September.
    7. Marco Avellaneda & Robert Buff & Craig Friedman & Nicolas Grandechamp & Lukasz Kruk & Joshua Newman, 2001. "Weighted Monte Carlo: A New Technique For Calibrating Asset-Pricing Models," International Journal of Theoretical and Applied Finance (IJTAF), World Scientific Publishing Co. Pte. Ltd., vol. 4(01), pages 91-119.
    8. Kleijnen, Jack P. C., 1995. "Verification and validation of simulation models," European Journal of Operational Research, Elsevier, vol. 82(1), pages 145-162, April.
    9. Henry Lam, 2018. "Sensitivity to Serial Dependency of Input Processes: A Robust Approach," Management Science, INFORMS, vol. 64(3), pages 1311-1327, March.
    10. Andrew E. B. Lim & J. George Shanthikumar, 2007. "Relative Entropy, Exponential Utility, and Robust Dynamic Pricing," Operations Research, INFORMS, vol. 55(2), pages 198-214, April.
    11. Henry Lam & Clementine Mottet, 2017. "Tail Analysis Without Parametric Models: A Worst-Case Perspective," Operations Research, INFORMS, vol. 65(6), pages 1696-1711, December.
    12. Yuhong Xu, 2014. "Robust valuation and risk measurement under model uncertainty," Papers 1407.8024, arXiv.org.
    13. D. J. Daley & L. D. Servi, 1998. "Moment estimation of customer loss rates from transactional data," International Journal of Stochastic Analysis, Hindawi, vol. 11, pages 1-10, January.
    14. Soumyadip Ghosh & Henry Lam, 2019. "Robust Analysis in Stochastic Simulation: Computation and Performance Guarantees," Operations Research, INFORMS, vol. 67(1), pages 232-249, January.
    15. Erick Delage & Yinyu Ye, 2010. "Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems," Operations Research, INFORMS, vol. 58(3), pages 595-612, June.
    16. Wolfram Wiesemann & Daniel Kuhn & Melvyn Sim, 2014. "Distributionally Robust Convex Optimization," Operations Research, INFORMS, vol. 62(6), pages 1358-1376, December.
    17. Paul Glasserman & Bin Yu, 2005. "Large Sample Properties of Weighted Monte Carlo Estimators," Operations Research, INFORMS, vol. 53(2), pages 298-312, April.
    18. Whitt, Ward, 2012. "Fitting birth-and-death queueing models to data," Statistics & Probability Letters, Elsevier, vol. 82(5), pages 998-1004.
    19. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    20. Garud N. Iyengar, 2005. "Robust Dynamic Programming," Mathematics of Operations Research, INFORMS, vol. 30(2), pages 257-280, May.
    21. Henry Lam, 2016. "Robust Sensitivity Analysis for Stochastic Systems," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1248-1275, November.
    22. Huan Xu & Shie Mannor, 2012. "Distributionally Robust Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 37(2), pages 288-300, May.
    23. Bernd Kraan & Tim Bedford, 2005. "Probabilistic Inversion of Expert Judgments in the Quantification of Model Uncertainty," Management Science, INFORMS, vol. 51(6), pages 995-1006, June.
    24. Mark Broadie & Deniz Cicek & Assaf Zeevi, 2011. "General Bounds and Finite-Time Improvement for the Kiefer-Wolfowitz Stochastic Approximation Algorithm," Operations Research, INFORMS, vol. 59(5), pages 1211-1224, October.
    25. Paul Glasserman & Linan Yang, 2018. "Bounding Wrong†Way Risk In Cva Calculation," Mathematical Finance, Wiley Blackwell, vol. 28(1), pages 268-305, January.
    26. Richard C. Larson, 1990. "The Queue Inference Engine: Deducing Queue Statistics from Transactional Data," Management Science, INFORMS, vol. 36(5), pages 586-601, May.
    27. Henry Lam, 2019. "Recovering Best Statistical Guarantees via the Empirical Divergence-Based Distributionally Robust Optimization," Operations Research, INFORMS, vol. 67(4), pages 1090-1105, July.
    28. Stephen E. Chick, 2001. "Input Distribution Selection for Simulation Experiments: Accounting for Input Uncertainty," Operations Research, INFORMS, vol. 49(5), pages 744-758, October.
    29. Marco Avellaneda & Robert Buff & Craig Friedman & Nicolas Grandechamp & Lukasz Kruk & Joshua Newman, 2001. "Weighted Monte Carlo: A New Technique For Calibrating Asset-Pricing Models," World Scientific Book Chapters, in: Marco Avellaneda (ed.), Quantitative Analysis In Financial Markets Collected Papers of the New York University Mathematical Finance Seminar(Volume II), chapter 9, pages 239-265, World Scientific Publishing Co. Pte. Ltd..
    30. Joel Goh & Melvyn Sim, 2010. "Distributionally Robust Optimization and Its Tractable Approximations," Operations Research, INFORMS, vol. 58(4-part-1), pages 902-917, August.
    31. Jose Blanchet & Karthyek Murthy, 2019. "Quantifying Distributional Model Risk via Optimal Transport," Mathematics of Operations Research, INFORMS, vol. 44(2), pages 565-600, May.
    32. Peter Hall & Juhyun Park, 2004. "Nonparametric inference about service time distribution from indirect measurements," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 66(4), pages 861-875, November.
    33. Paul Glasserman & Xingbo Xu, 2014. "Robust risk measurement and model risk," Quantitative Finance, Taylor & Francis Journals, vol. 14(1), pages 29-58, January.
    34. Grani A. Hanasusanto & Vladimir Roitch & Daniel Kuhn & Wolfram Wiesemann, 2017. "Ambiguous Joint Chance Constraints Under Mean and Dispersion Information," Operations Research, INFORMS, vol. 65(3), pages 751-767, June.
    35. Zhaolin Hu & Jing Cao & L. Jeff Hong, 2012. "Robust Simulation of Global Warming Policies Using the DICE Model," Management Science, INFORMS, vol. 58(12), pages 2190-2206, December.
    36. N. Bingham & Susan Pitts, 1999. "Non-parametric Estimation for the M/G/∞ Queue," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 51(1), pages 71-97, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yijie Peng & Michael C. Fu & Bernd Heidergott & Henry Lam, 2020. "Maximum Likelihood Estimation by Monte Carlo Simulation: Toward Data-Driven Stochastic Modeling," Operations Research, INFORMS, vol. 68(6), pages 1896-1912, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Soumyadip Ghosh & Henry Lam, 2019. "Robust Analysis in Stochastic Simulation: Computation and Performance Guarantees," Operations Research, INFORMS, vol. 67(1), pages 232-249, January.
    2. Jose Blanchet & Karthyek Murthy, 2019. "Quantifying Distributional Model Risk via Optimal Transport," Mathematics of Operations Research, INFORMS, vol. 44(2), pages 565-600, May.
    3. Zhaolin Hu & L. Jeff Hong, 2022. "Robust Simulation with Likelihood-Ratio Constrained Input Uncertainty," INFORMS Journal on Computing, INFORMS, vol. 34(4), pages 2350-2367, July.
    4. L. Jeff Hong & Zhiyuan Huang & Henry Lam, 2021. "Learning-Based Robust Optimization: Procedures and Statistical Guarantees," Management Science, INFORMS, vol. 67(6), pages 3447-3467, June.
    5. Henry Lam, 2018. "Sensitivity to Serial Dependency of Input Processes: A Robust Approach," Management Science, INFORMS, vol. 64(3), pages 1311-1327, March.
    6. Henry Lam, 2016. "Robust Sensitivity Analysis for Stochastic Systems," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1248-1275, November.
    7. Henry Lam, 2019. "Recovering Best Statistical Guarantees via the Empirical Divergence-Based Distributionally Robust Optimization," Operations Research, INFORMS, vol. 67(4), pages 1090-1105, July.
    8. Henry Lam & Clementine Mottet, 2017. "Tail Analysis Without Parametric Models: A Worst-Case Perspective," Operations Research, INFORMS, vol. 65(6), pages 1696-1711, December.
    9. Andrew J. Keith & Darryl K. Ahner, 2021. "A survey of decision making and optimization under uncertainty," Annals of Operations Research, Springer, vol. 300(2), pages 319-353, May.
    10. Feng Liu & Zhi Chen & Shuming Wang, 2023. "Globalized Distributionally Robust Counterpart," INFORMS Journal on Computing, INFORMS, vol. 35(5), pages 1120-1142, September.
    11. Zhi Chen & Weijun Xie, 2021. "Regret in the Newsvendor Model with Demand and Yield Randomness," Production and Operations Management, Production and Operations Management Society, vol. 30(11), pages 4176-4197, November.
    12. Zhi Chen & Melvyn Sim & Huan Xu, 2019. "Distributionally Robust Optimization with Infinitely Constrained Ambiguity Sets," Operations Research, INFORMS, vol. 67(5), pages 1328-1344, September.
    13. Vishal Gupta, 2019. "Near-Optimal Bayesian Ambiguity Sets for Distributionally Robust Optimization," Management Science, INFORMS, vol. 65(9), pages 4242-4260, September.
    14. Kim, Sojung & Weber, Stefan, 2022. "Simulation methods for robust risk assessment and the distorted mix approach," European Journal of Operational Research, Elsevier, vol. 298(1), pages 380-398.
    15. Liu, Haiyan & Mao, Tiantian, 2022. "Distributionally robust reinsurance with Value-at-Risk and Conditional Value-at-Risk," Insurance: Mathematics and Economics, Elsevier, vol. 107(C), pages 393-417.
    16. Wolfram Wiesemann & Daniel Kuhn & Melvyn Sim, 2014. "Distributionally Robust Convex Optimization," Operations Research, INFORMS, vol. 62(6), pages 1358-1376, December.
    17. van Eekelen, Wouter, 2023. "Distributionally robust views on queues and related stochastic models," Other publications TiSEM 9b99fc05-9d68-48eb-ae8c-9, Tilburg University, School of Economics and Management.
    18. Haolin Ruan & Zhi Chen & Chin Pang Ho, 2023. "Adjustable Distributionally Robust Optimization with Infinitely Constrained Ambiguity Sets," INFORMS Journal on Computing, INFORMS, vol. 35(5), pages 1002-1023, September.
    19. Corlu, Canan G. & Akcay, Alp & Xie, Wei, 2020. "Stochastic simulation under input uncertainty: A Review," Operations Research Perspectives, Elsevier, vol. 7(C).
    20. Viet Anh Nguyen & Soroosh Shafiee & Damir Filipovi'c & Daniel Kuhn, 2021. "Mean-Covariance Robust Risk Measurement," Papers 2112.09959, arXiv.org, revised Nov 2023.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:67:y:2019:i:5:p:1362-1382. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.