IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2310.07052.html
   My bibliography  Save this paper

Uses of Sub-sample Estimates to Reduce Errors in Stochastic Optimization Models

Author

Listed:
  • John R. Birge

Abstract

Optimization software enables the solution of problems with millions of variables and associated parameters. These parameters are, however, often uncertain and represented with an analytical description of the parameter's distribution or with some form of sample. With large numbers of such parameters, optimization of the resulting model is often driven by mis-specifications or extreme sample characteristics, resulting in solutions that are far from a true optimum. This paper describes how asymptotic convergence results may not be useful in large-scale problems and how the optimization of problems based on sub-sample estimates may achieve improved results over models using full-sample solution estimates. A motivating example and numerical results from a portfolio optimization problem demonstrate the potential improvement. A theoretical analysis also provides insight into the structure of problems where sub-sample optimization may be most beneficial.

Suggested Citation

  • John R. Birge, 2023. "Uses of Sub-sample Estimates to Reduce Errors in Stochastic Optimization Models," Papers 2310.07052, arXiv.org.
  • Handle: RePEc:arx:papers:2310.07052
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2310.07052
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ravi Jagannathan & Tongshu Ma, 2003. "Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps," Journal of Finance, American Finance Association, vol. 58(4), pages 1651-1683, August.
    2. R. W. Conway, 1963. "Some Tactical Problems in Digital Simulation," Management Science, INFORMS, vol. 10(1), pages 47-61, October.
    3. Kan, Raymond & Zhou, Guofu, 2007. "Optimal Portfolio Choice with Parameter Uncertainty," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 42(3), pages 621-656, September.
    4. Andreas Eichhorn & Werner Römisch, 2007. "Stochastic Integer Programming: Limit Theorems and Confidence Intervals," Mathematics of Operations Research, INFORMS, vol. 32(1), pages 118-135, February.
    5. George S. Fishman, 1978. "Grouping Observations in Digital Simulation," Management Science, INFORMS, vol. 24(5), pages 510-521, January.
    6. Vishal Gupta & Paat Rusmevichientong, 2021. "Small-Data, Large-Scale Linear Optimization with Uncertain Objectives," Management Science, INFORMS, vol. 67(1), pages 220-241, January.
    7. Averill M. Law & John S. Carson, 1979. "A Sequential Procedure for Determining the Length of a Steady-State Simulation," Operations Research, INFORMS, vol. 27(5), pages 1011-1025, October.
    8. repec:bla:jfinan:v:58:y:2003:i:4:p:1651-1684 is not listed on IDEAS
    9. Natalie M. Steiger & James R. Wilson, 2001. "Convergence Properties of the Batch Means Method for Simulation Output Analysis," INFORMS Journal on Computing, INFORMS, vol. 13(4), pages 277-293, November.
    10. Donald W. K. Andrews, 2000. "Inconsistency of the Bootstrap when a Parameter Is on the Boundary of the Parameter Space," Econometrica, Econometric Society, vol. 68(2), pages 399-406, March.
    11. L. Dai & C. H. Chen & J. R. Birge, 2000. "Convergence Properties of Two-Stage Stochastic Programming," Journal of Optimization Theory and Applications, Springer, vol. 106(3), pages 489-509, September.
    12. James E. Smith & Robert L. Winkler, 2006. "The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis," Management Science, INFORMS, vol. 52(3), pages 311-322, March.
    13. Robert E. Bixby, 2002. "Solving Real-World Linear Programs: A Decade and More of Progress," Operations Research, INFORMS, vol. 50(1), pages 3-15, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Natalie M. Steiger & James R. Wilson, 2002. "An Improved Batch Means Procedure for Simulation Output Analysis," Management Science, INFORMS, vol. 48(12), pages 1569-1586, December.
    2. Song, Wheyming T. & Chih, Mingchang, 2010. "Extended dynamic partial-overlapping batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 203(3), pages 640-651, June.
    3. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2020. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," LEM Papers Series 2020/31, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    4. Song, Wheyming Tina & Chih, Mingchang, 2013. "Run length not required: Optimal-mse dynamic batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 229(1), pages 114-123.
    5. Vandin, Andrea & Giachini, Daniele & Lamperti, Francesco & Chiaromonte, Francesca, 2022. "Automated and distributed statistical analysis of economic agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 143(C).
    6. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2021. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," Papers 2102.05405, arXiv.org, revised Nov 2023.
    7. Lada, Emily K. & Wilson, James R., 2006. "A wavelet-based spectral procedure for steady-state simulation analysis," European Journal of Operational Research, Elsevier, vol. 174(3), pages 1769-1801, November.
    8. Enver Yücesan, 1993. "Randomization tests for initialization bias in simulation output," Naval Research Logistics (NRL), John Wiley & Sons, vol. 40(5), pages 643-663, August.
    9. Wang, Christina Dan & Chen, Zhao & Lian, Yimin & Chen, Min, 2022. "Asset selection based on high frequency Sharpe ratio," Journal of Econometrics, Elsevier, vol. 227(1), pages 168-188.
    10. Füss, Roland & Miebs, Felix & Trübenbach, Fabian, 2014. "A jackknife-type estimator for portfolio revision," Journal of Banking & Finance, Elsevier, vol. 43(C), pages 14-28.
    11. Francisco Peñaranda & Enrique Sentana, 2024. "Portfolio management with big data," Working Papers wp2024_2411, CEMFI.
    12. Hautsch, Nikolaus & Voigt, Stefan, 2019. "Large-scale portfolio allocation under transaction costs and model uncertainty," Journal of Econometrics, Elsevier, vol. 212(1), pages 221-240.
    13. Hiraki, Kazuhiro & Sun, Chuanping, 2022. "A toolkit for exploiting contemporaneous stock correlations," Journal of Empirical Finance, Elsevier, vol. 65(C), pages 99-124.
    14. Michele Costola & Bertrand Maillet & Zhining Yuan & Xiang Zhang, 2024. "Mean–variance efficient large portfolios: a simple machine learning heuristic technique based on the two-fund separation theorem," Annals of Operations Research, Springer, vol. 334(1), pages 133-155, March.
    15. Istvan Varga-Haszonits & Fabio Caccioli & Imre Kondor, 2016. "Replica approach to mean-variance portfolio optimization," Papers 1606.08679, arXiv.org.
    16. Serge Darolles & Christian Gouriéroux & Emmanuelle Jay, 2012. "Robust Portfolio Allocation with Systematic Risk Contribution Restrictions," Working Papers 2012-35, Center for Research in Economics and Statistics.
    17. Thomas J. Brennan & Andrew W. Lo, 2010. "Impossible Frontiers," Management Science, INFORMS, vol. 56(6), pages 905-923, June.
    18. Jonathan Fletcher, 2009. "Risk Reduction and Mean‐Variance Analysis: An Empirical Investigation," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 36(7‐8), pages 951-971, September.
    19. David Allen & Stephen Satchell & Colin Lizieri, 2024. "Quantifying the non-Gaussian gain," Journal of Asset Management, Palgrave Macmillan, vol. 25(1), pages 1-18, February.
    20. Thomas Trier Bjerring & Omri Ross & Alex Weissensteiner, 2017. "Feature selection for portfolio optimization," Annals of Operations Research, Springer, vol. 256(1), pages 21-40, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2310.07052. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.