IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v65y2017i3p821-836.html
   My bibliography  Save this article

Efficient Ranking and Selection in Parallel Computing Environments

Author

Listed:
  • Eric C. Ni

    (School of Operations Research and Information Engineering, Cornell University, Ithaca, New York 14853)

  • Dragos F. Ciocan

    (Technology and Operations Management, INSEAD, Fontainebleau, France 77300)

  • Shane G. Henderson

    (School of Operations Research and Information Engineering, Cornell University, Ithaca, New York 14853)

  • Susan R. Hunter

    (School of Industrial Engineering, Purdue University, West Lafayette, Indiana 47907)

Abstract

The goal of ranking and selection (R&S) procedures is to identify the best stochastic system from among a finite set of competing alternatives. Such procedures require constructing estimates of each system’s performance, which can be obtained simultaneously by running multiple independent replications on a parallel computing platform. Nontrivial statistical and implementation issues arise when designing R&S procedures for a parallel computing environment. We propose several design principles for parallel R&S procedures that preserve statistical validity and maximize core utilization, especially when large numbers of alternatives or cores are involved. These principles are followed closely by our parallel Good Selection Procedure (GSP), which, under the assumption of normally distributed output, (i) guarantees to select a system in the indifference zone with high probability, (ii) in tests on up to 1,024 parallel cores runs efficiently, and (iii) in an example uses smaller sample sizes compared to existing parallel procedures, particularly for large problems (over 10 6 alternatives). In our computational study we discuss three methods for implementing GSP on parallel computers, namely the Message-Passing Interface (MPI), Hadoop MapReduce, and Spark, and show that Spark provides a good compromise between the efficiency of MPI and robustness to core failures.

Suggested Citation

  • Eric C. Ni & Dragos F. Ciocan & Shane G. Henderson & Susan R. Hunter, 2017. "Efficient Ranking and Selection in Parallel Computing Environments," Operations Research, INFORMS, vol. 65(3), pages 821-836, June.
    • Handle: RePEc:inm:oropre:v:65:y:2017:i:3:p:821-836
    DOI: 10.1287/opre.2016.1577
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/opre.2016.1577
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.2016.1577?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Pierre L'Ecuyer & Richard Simard & E. Jack Chen & W. David Kelton, 2002. "An Object-Oriented Random-Number Package with Many Long Streams and Substreams," Operations Research, INFORMS, vol. 50(6), pages 1073-1075, December.
    2. Michael C. Fu, 2002. "Feature Article: Optimization for simulation: Theory vs. Practice," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 192-215, August.
    3. Barry L. Nelson & Julie Swann & David Goldsman & Wheyming Song, 2001. "Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large," Operations Research, INFORMS, vol. 49(6), pages 950-963, December.
    4. Justin Boesel & Barry L. Nelson & Seong-Hee Kim, 2003. "Using Ranking and Selection to “Clean Up” after Simulation Optimization," Operations Research, INFORMS, vol. 51(5), pages 814-825, October.
    5. Chun-Hung Chen & Stephen E. Chick & Loo Hay Lee & Nugroho A. Pujowidianto, 2015. "Ranking and Selection: Efficient Simulation Budget Allocation," International Series in Operations Research & Management Science, in: Michael C Fu (ed.), Handbook of Simulation Optimization, edition 127, chapter 0, pages 45-80, Springer.
    6. Peter W. Glynn & Philip Heidelberger, 1990. "Bias Properties of Budget Constrained Simulations," Operations Research, INFORMS, vol. 38(5), pages 801-814, October.
    7. Jun Luo & L. Jeff Hong & Barry L. Nelson & Yang Wu, 2015. "Fully Sequential Procedures for Large-Scale Ranking-and-Selection Problems in Parallel Computing Environments," Operations Research, INFORMS, vol. 63(5), pages 1177-1194, October.
    8. Barry L. Nelson & Frank J. Matejcik, 1995. "Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in Simulation," Management Science, INFORMS, vol. 41(12), pages 1935-1945, December.
    9. L. Jeff Hong, 2006. "Fully sequential indifference‐zone selection procedures with variance‐dependent sampling," Naval Research Logistics (NRL), John Wiley & Sons, vol. 53(5), pages 464-476, August.
    10. Andrew Karl & Randy Eubank & Jelena Milovanovic & Mark Reiser & Dennis Young, 2014. "Using RngStreams for parallel random number generation in C++ and R," Computational Statistics, Springer, vol. 29(5), pages 1301-1320, October.
    11. Peter W. Glynn & Ward Whitt, 1992. "The Asymptotic Efficiency of Simulation Estimators," Operations Research, INFORMS, vol. 40(3), pages 505-520, June.
    12. Pichitlamken, Juta & Nelson, Barry L. & Hong, L. Jeff, 2006. "A sequential procedure for neighborhood selection-of-the-best in optimization via simulation," European Journal of Operational Research, Elsevier, vol. 173(1), pages 283-298, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ying Zhong & Shaoxuan Liu & Jun Luo & L. Jeff Hong, 2022. "Speeding Up Paulson’s Procedure for Large-Scale Problems Using Parallel Computing," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 586-606, January.
    2. L. Jeff Hong & Guangxin Jiang & Ying Zhong, 2022. "Solving Large-Scale Fixed-Budget Ranking and Selection Problems," INFORMS Journal on Computing, INFORMS, vol. 34(6), pages 2930-2949, November.
    3. Taylor, Simon J.E., 2019. "Distributed simulation: state-of-the-art and potential for operational research," European Journal of Operational Research, Elsevier, vol. 273(1), pages 1-19.
    4. Saeid Delshad & Amin Khademi, 2020. "Information theory for ranking and selection," Naval Research Logistics (NRL), John Wiley & Sons, vol. 67(4), pages 239-253, June.
    5. Zhongshun Shi & Yijie Peng & Leyuan Shi & Chun-Hung Chen & Michael C. Fu, 2022. "Dynamic Sampling Allocation Under Finite Simulation Budget for Feasibility Determination," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 557-568, January.
    6. Weiwei Fan & L. Jeff Hong & Xiaowei Zhang, 2020. "Distributionally Robust Selection of the Best," Management Science, INFORMS, vol. 66(1), pages 190-208, January.
    7. Daniel Russo, 2020. "Simple Bayesian Algorithms for Best-Arm Identification," Operations Research, INFORMS, vol. 68(6), pages 1625-1647, November.
    8. David J. Eckman & Shane G. Henderson, 2022. "Posterior-Based Stopping Rules for Bayesian Ranking-and-Selection Procedures," INFORMS Journal on Computing, INFORMS, vol. 34(3), pages 1711-1728, May.
    9. Yishuang Hu & Yi Ding & Zhiguo Zeng, 2022. "Redundancy optimization for multi-state series-parallel systems using ordinal optimization-based-genetic algorithm," Journal of Risk and Reliability, , vol. 236(1), pages 66-78, February.
    10. Haihui Shen & L. Jeff Hong & Xiaowei Zhang, 2021. "Ranking and Selection with Covariates for Personalized Decision Making," INFORMS Journal on Computing, INFORMS, vol. 33(4), pages 1500-1519, October.
    11. Ruijing Wu & Shaoxuan Liu & Zhenyang Shi, 2019. "Customer Incentive Rebalancing Plan in Free-Float Bike-Sharing System with Limited Information," Sustainability, MDPI, vol. 11(11), pages 1-24, May.
    12. Cheng Li & Siyang Gao & Jianzhong Du, 2023. "Convergence Analysis of Stochastic Kriging-Assisted Simulation with Random Covariates," INFORMS Journal on Computing, INFORMS, vol. 35(2), pages 386-402, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ying Zhong & Shaoxuan Liu & Jun Luo & L. Jeff Hong, 2022. "Speeding Up Paulson’s Procedure for Large-Scale Problems Using Parallel Computing," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 586-606, January.
    2. Tsai, Shing Chih, 2011. "Selecting the best simulated system with weighted control-variate estimators," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 82(4), pages 705-717.
    3. Shing Chih Tsai, 2013. "Rapid Screening Procedures for Zero-One Optimization via Simulation," INFORMS Journal on Computing, INFORMS, vol. 25(2), pages 317-331, May.
    4. Haihui Shen & L. Jeff Hong & Xiaowei Zhang, 2021. "Ranking and Selection with Covariates for Personalized Decision Making," INFORMS Journal on Computing, INFORMS, vol. 33(4), pages 1500-1519, October.
    5. Shing Chih Tsai & Tse Yang, 2017. "Rapid screening algorithms for stochastically constrained problems," Annals of Operations Research, Springer, vol. 254(1), pages 425-447, July.
    6. David J. Eckman & Shane G. Henderson, 2022. "Posterior-Based Stopping Rules for Bayesian Ranking-and-Selection Procedures," INFORMS Journal on Computing, INFORMS, vol. 34(3), pages 1711-1728, May.
    7. Shing Chih Tsai & Chen Hao Kuo, 2012. "Screening and selection procedures with control variates and correlation induction techniques," Naval Research Logistics (NRL), John Wiley & Sons, vol. 59(5), pages 340-361, August.
    8. Jun Luo & L. Jeff Hong & Barry L. Nelson & Yang Wu, 2015. "Fully Sequential Procedures for Large-Scale Ranking-and-Selection Problems in Parallel Computing Environments," Operations Research, INFORMS, vol. 63(5), pages 1177-1194, October.
    9. Miguel Lejeune & François Margot, 2011. "Optimization for simulation: LAD accelerator," Annals of Operations Research, Springer, vol. 188(1), pages 285-305, August.
    10. M Laguna & J Molina & F Pérez & R Caballero & A G Hernández-Díaz, 2010. "The challenge of optimizing expensive black boxes: a scatter search/rough set theory approach," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 61(1), pages 53-67, January.
    11. Justin Boesel & Barry L. Nelson & Seong-Hee Kim, 2003. "Using Ranking and Selection to “Clean Up” after Simulation Optimization," Operations Research, INFORMS, vol. 51(5), pages 814-825, October.
    12. Weiwei Fan & L. Jeff Hong & Barry L. Nelson, 2016. "Indifference-Zone-Free Selection of the Best," Operations Research, INFORMS, vol. 64(6), pages 1499-1514, December.
    13. Sigrún Andradóttir & Seong‐Hee Kim, 2010. "Fully sequential procedures for comparing constrained systems via simulation," Naval Research Logistics (NRL), John Wiley & Sons, vol. 57(5), pages 403-421, August.
    14. Michael B. Gordy & Sandeep Juneja, 2010. "Nested Simulation in Portfolio Risk Measurement," Management Science, INFORMS, vol. 56(10), pages 1833-1848, October.
    15. L. Jeff Hong & Barry L. Nelson, 2006. "Discrete Optimization via Simulation Using COMPASS," Operations Research, INFORMS, vol. 54(1), pages 115-129, February.
    16. Sigrún Andradóttir, 2002. "Simulation Optimization: Integrating Research and Practice," INFORMS Journal on Computing, INFORMS, vol. 14(3), pages 216-219, August.
    17. Tsai, Shing Chih & Chen, Sin Ting, 2017. "A simulation-based multi-objective optimization framework: A case study on inventory management," Omega, Elsevier, vol. 70(C), pages 148-159.
    18. Huashuai Qu & Ilya O. Ryzhov & Michael C. Fu & Zi Ding, 2015. "Sequential Selection with Unknown Correlation Structures," Operations Research, INFORMS, vol. 63(4), pages 931-948, August.
    19. Shing Chih Tsai & Wu Hung Lin & Chia Cheng Wu & Shao Jen Weng & Ching Fen Tang, 2022. "Decision support algorithms for optimizing surgery start times considering the performance variation," Health Care Management Science, Springer, vol. 25(2), pages 208-221, June.
    20. Tsai, Shing Chih & Chu, I-Hao, 2012. "Controlled multistage selection procedures for comparison with a standard," European Journal of Operational Research, Elsevier, vol. 223(3), pages 709-721.

    More about this item

    Keywords

    ranking and selection; parallel computing;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:65:y:2017:i:3:p:821-836. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.