IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v307y2023i1p266-278.html

On the finite-sample statistical validity of adaptive fully sequential procedures

Author

Listed:
  • Cheng, Zhenxia
  • Luo, Jun
  • Wu, Ruijing

Abstract

We consider the simulation optimization problem of selecting the best system design from a finite set of alternatives, which is known as ranking and selection (R&S). Many fully sequential procedures have been proposed to solve the R&S problem using a static sampling rule in order to ensure a finite-sample statistical guarantee. In this paper, we develop fully sequential procedures that can incorporate various adaptive sampling rules, based on a modification of Paulson’s bound Paulson (1964), while still preserving the finite-sample guarantee. In particular, we propose an adaptive sampling rule that utilizes the consecutively updated sample mean and sample variance information by solving a minimization problem of the approximated total sample size. Finally, we demonstrate the efficiency of the proposed procedures with several existing procedures through extensive simulation experiments, and apply them to solve an ambulance dispatching problem.

Suggested Citation

  • Cheng, Zhenxia & Luo, Jun & Wu, Ruijing, 2023. "On the finite-sample statistical validity of adaptive fully sequential procedures," European Journal of Operational Research, Elsevier, vol. 307(1), pages 266-278.
    • Handle: RePEc:eee:ejores:v:307:y:2023:i:1:p:266-278
    DOI: 10.1016/j.ejor.2022.11.038
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221722008955
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2022.11.038?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    2. Giulia Pedrielli & K. Selcuk Candan & Xilun Chen & Logan Mathesen & Alireza Inanalouganji & Jie Xu & Chun-Hung Chen & Loo Hay Lee, 2019. "Generalized Ordinal Learning Framework (GOLF) for Decision Making with Future Simulated Data," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 36(06), pages 1-35, December.
    3. Susan R. Hunter & Benjamin McClosky, 2016. "Maximizing quantitative traits in the mating design problem via simulation-based Pareto estimation," IISE Transactions, Taylor & Francis Journals, vol. 48(6), pages 565-578, June.
    4. Batur, D. & Choobineh, F., 2010. "A quantile-based approach to system selection," European Journal of Operational Research, Elsevier, vol. 202(3), pages 764-772, May.
    5. Weiwei Fan & L. Jeff Hong & Barry L. Nelson, 2016. "Indifference-Zone-Free Selection of the Best," Operations Research, INFORMS, vol. 64(6), pages 1499-1514, December.
    6. Tsai, Shing Chih & Chen, Sin Ting, 2017. "A simulation-based multi-objective optimization framework: A case study on inventory management," Omega, Elsevier, vol. 70(C), pages 148-159.
    7. Yijie Peng & Chun-Hung Chen & Michael C. Fu & Jian-Qiang Hu & Ilya O. Ryzhov, 2021. "Efficient Sampling Allocation Procedures for Optimal Quantile Selection," INFORMS Journal on Computing, INFORMS, vol. 33(1), pages 230-245, January.
    8. Lee, Mi Lim & Park, Chuljin & Park, Dong Uk, 2018. "Self-adjusting the tolerance level in a fully sequential feasibility check procedure," European Journal of Operational Research, Elsevier, vol. 271(2), pages 733-745.
    9. Daniel Russo, 2020. "Simple Bayesian Algorithms for Best-Arm Identification," Operations Research, INFORMS, vol. 68(6), pages 1625-1647, November.
    10. L. Jeff Hong, 2006. "Fully sequential indifference‐zone selection procedures with variance‐dependent sampling," Naval Research Logistics (NRL), John Wiley & Sons, vol. 53(5), pages 464-476, August.
    11. Ruijing Wu & Shaoxuan Liu & Zhenyang Shi, 2020. "Algorithm for Calculating the Initial Sample Size in a Fully Sequential Ranking and Selection Procedure," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 37(03), pages 1-19, May.
    12. Stephen E. Chick & Koichiro Inoue, 2001. "New Two-Stage and Sequential Procedures for Selecting the Best Simulated System," Operations Research, INFORMS, vol. 49(5), pages 732-743, October.
    13. Siyang Gao & Weiwei Chen & Leyuan Shi, 2017. "A New Budget Allocation Framework for the Expected Opportunity Cost," Operations Research, INFORMS, vol. 65(3), pages 787-803, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cao, Zirui & Wang, Haowei & Chew, Ek Peng & Li, Haobin & Tan, Kok Choon, 2025. "A budget-adaptive allocation rule for optimal computing budget allocation," European Journal of Operational Research, Elsevier, vol. 325(2), pages 247-260.
    2. Tsai, Shing Chih & Jiang, Guangxin, 2025. "Adaptive sequential selection procedures for optimal quantile with control variates," European Journal of Operational Research, Elsevier, vol. 326(3), pages 515-529.
    3. Huang, Hao & Tsai, Shing Chih & Park, Chuljin, 2025. "Probabilistic branch and bound considering stochastic constraints," European Journal of Operational Research, Elsevier, vol. 321(1), pages 147-159.
    4. Li, Dongyang & Chew, Ek Peng & Li, Haobin & Yücesan, Enver & Chen, Chun-Hung, 2026. "Efficient simulation budget allocation for contextual ranking and selection with quadratic models," European Journal of Operational Research, Elsevier, vol. 328(3), pages 862-876.
    5. Zhong, Ying & Du, Jianzhong & Li, Deng-Feng & Hu, Zhaolin, 2025. "Reference alternatives based knockout-tournament procedure for ranking and selection," European Journal of Operational Research, Elsevier, vol. 320(3), pages 628-641.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhong, Ying & Du, Jianzhong & Li, Deng-Feng & Hu, Zhaolin, 2025. "Reference alternatives based knockout-tournament procedure for ranking and selection," European Journal of Operational Research, Elsevier, vol. 320(3), pages 628-641.
    2. Zhongshun Shi & Yijie Peng & Leyuan Shi & Chun-Hung Chen & Michael C. Fu, 2022. "Dynamic Sampling Allocation Under Finite Simulation Budget for Feasibility Determination," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 557-568, January.
    3. Zaile Li & Weiwei Fan & L. Jeff Hong, 2025. "The (Surprising) Sample Optimality of Greedy Procedures for Large-Scale Ranking and Selection," Management Science, INFORMS, vol. 71(2), pages 1238-1259, February.
    4. Ye Chen & Ilya O. Ryzhov, 2023. "Balancing Optimal Large Deviations in Sequential Selection," Management Science, INFORMS, vol. 69(6), pages 3457-3473, June.
    5. Linda Pei & Barry L. Nelson & Susan R. Hunter, 2024. "Parallel Adaptive Survivor Selection," Operations Research, INFORMS, vol. 72(1), pages 336-354, January.
    6. Haihui Shen & L. Jeff Hong & Xiaowei Zhang, 2021. "Ranking and Selection with Covariates for Personalized Decision Making," INFORMS Journal on Computing, INFORMS, vol. 33(4), pages 1500-1519, October.
    7. Li, Dongyang & Chew, Ek Peng & Li, Haobin & Yücesan, Enver & Chen, Chun-Hung, 2026. "Efficient simulation budget allocation for contextual ranking and selection with quadratic models," European Journal of Operational Research, Elsevier, vol. 328(3), pages 862-876.
    8. Ying Zhong & L. Jeff Hong, 2022. "Knockout-Tournament Procedures for Large-Scale Ranking and Selection in Parallel Computing Environments," Operations Research, INFORMS, vol. 70(1), pages 432-453, January.
    9. Tsai, Shing Chih & Jiang, Guangxin, 2025. "Adaptive sequential selection procedures for optimal quantile with control variates," European Journal of Operational Research, Elsevier, vol. 326(3), pages 515-529.
    10. Zhongshun Shi & Siyang Gao & Hui Xiao & Weiwei Chen, 2019. "A worst‐case formulation for constrained ranking and selection with input uncertainty," Naval Research Logistics (NRL), John Wiley & Sons, vol. 66(8), pages 648-662, December.
    11. Demet Batur & F. Fred Choobineh, 2021. "Selecting the Best Alternative Based on Its Quantile," INFORMS Journal on Computing, INFORMS, vol. 33(2), pages 657-671, May.
    12. Weiwei Fan & Xuewen Li & Jun Luo & Shing Chih Tsai, 2025. "Technical Note—First-Stage Sampling in Ranking and Selection: Beyond Variance Estimation," Operations Research, INFORMS, vol. 73(6), pages 3333-3344, November.
    13. Cao, Zirui & Wang, Haowei & Chew, Ek Peng & Li, Haobin & Tan, Kok Choon, 2025. "A budget-adaptive allocation rule for optimal computing budget allocation," European Journal of Operational Research, Elsevier, vol. 325(2), pages 247-260.
    14. Peter I. Frazier, 2014. "A Fully Sequential Elimination Procedure for Indifference-Zone Ranking and Selection with Tight Bounds on Probability of Correct Selection," Operations Research, INFORMS, vol. 62(4), pages 926-942, August.
    15. Jianzhong Du & Siyang Gao & Chun-Hung Chen, 2024. "A Contextual Ranking and Selection Method for Personalized Medicine," Manufacturing & Service Operations Management, INFORMS, vol. 26(1), pages 167-181, January.
    16. Wang, Tianxiang & Xu, Jie & Hu, Jian-Qiang & Chen, Chun-Hung, 2023. "Efficient estimation of a risk measure requiring two-stage simulation optimization," European Journal of Operational Research, Elsevier, vol. 305(3), pages 1355-1365.
    17. Chao Qin & Wei You, 2026. "Dual-Directed Algorithm Design for Efficient Pure Exploration," Operations Research, INFORMS, vol. 74(2), pages 1104-1125, March.
    18. Weiwei Fan & L. Jeff Hong & Barry L. Nelson, 2016. "Indifference-Zone-Free Selection of the Best," Operations Research, INFORMS, vol. 64(6), pages 1499-1514, December.
    19. Dongwook Shin & Mark Broadie & Assaf Zeevi, 2022. "Practical Nonparametric Sampling Strategies for Quantile-Based Ordinal Optimization," INFORMS Journal on Computing, INFORMS, vol. 34(2), pages 752-768, March.
    20. Gongbo Zhang & Yijie Peng & Jianghua Zhang & Enlu Zhou, 2023. "Asymptotically Optimal Sampling Policy for Selecting Top- m Alternatives," INFORMS Journal on Computing, INFORMS, vol. 35(6), pages 1261-1285, November.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:307:y:2023:i:1:p:266-278. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.