IDEAS home Printed from https://ideas.repec.org/a/inm/orijoc/v37y2025i4p785-807.html

An Efficient Node Selection Policy for Monte Carlo Tree Search with Neural Networks

Author

Listed:
  • Xiaotian Liu

    (PKU-Wuhan Institute for Artificial Intelligence, Guanghua School of Management, Peking University, Beijing 100871, China; and Xiangjiang Laboratory, Changsha 410000, China)

  • Yijie Peng

    (PKU-Wuhan Institute for Artificial Intelligence, Guanghua School of Management, Peking University, Beijing 100871, China; and Xiangjiang Laboratory, Changsha 410000, China)

  • Gongbo Zhang

    (PKU-Wuhan Institute for Artificial Intelligence, Guanghua School of Management, Peking University, Beijing 100871, China; and Xiangjiang Laboratory, Changsha 410000, China)

  • Ruihan Zhou

    (PKU-Wuhan Institute for Artificial Intelligence, Guanghua School of Management, Peking University, Beijing 100871, China; and Xiangjiang Laboratory, Changsha 410000, China)

Abstract

Monte Carlo tree search (MCTS) has been gaining increasing popularity, and the success of AlphaGo has prompted a new trend of incorporating a value network and a policy network constructed with neural networks into MCTS, namely, NN-MCTS. In this work, motivated by the shortcomings of the widely used upper confidence bounds applied to trees (UCT) policy, we formulate the node selection problem in NN-MCTS as a multistage ranking and selection (R&S) problem and propose a node selection policy that efficiently allocates a limited search budget to maximize the probability of correctly selecting the best action at the root state. The value and policy networks in NN-MCTS further improve the performance of the proposed node selection policy by providing prior knowledge and guiding the selection of the final action, respectively. Numerical experiments on two board games and an OpenAI task demonstrate that the proposed method outperforms the UCT policy used in AlphaGo Zero and MuZero, implying the potential of constructing node selection policies in NN-MCTS with R&S procedures.

Suggested Citation

  • Xiaotian Liu & Yijie Peng & Gongbo Zhang & Ruihan Zhou, 2025. "An Efficient Node Selection Policy for Monte Carlo Tree Search with Neural Networks," INFORMS Journal on Computing, INFORMS, vol. 37(4), pages 785-807, July.
    • Handle: RePEc:inm:orijoc:v:37:y:2025:i:4:p:785-807
    DOI: 10.1287/ijoc.2023.0307
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/ijoc.2023.0307
    Download Restriction: no
    File URL: https://libkey.io/10.1287/ijoc.2023.0307?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Richard Bellman, 1954. "Some Applications of the Theory of Dynamic Programming---A Review," Operations Research, INFORMS, vol. 2(3), pages 275-288, August.
    2. Haidong Li & Henry Lam & Yijie Peng, 2024. "Efficient Learning for Clustering and Optimizing Context-Dependent Designs," Operations Research, INFORMS, vol. 72(2), pages 617-638, March.
    3. Hyeong Soo Chang & Michael C. Fu & Jiaqiao Hu & Steven I. Marcus, 2005. "An Adaptive Sampling Algorithm for Solving Markov Decision Processes," Operations Research, INFORMS, vol. 53(1), pages 126-139, February.
    4. Julian Schrittwieser & Ioannis Antonoglou & Thomas Hubert & Karen Simonyan & Laurent Sifre & Simon Schmitt & Arthur Guez & Edward Lockhart & Demis Hassabis & Thore Graepel & Timothy Lillicrap & David , 2020. "Mastering Atari, Go, chess and shogi by planning with a learned model," Nature, Nature, vol. 588(7839), pages 604-609, December.
    5. Daniel Russo, 2020. "Simple Bayesian Algorithms for Best-Arm Identification," Operations Research, INFORMS, vol. 68(6), pages 1625-1647, November.
    6. Richard Bellman, 1954. "On some applications of the theory of dynamic programming to logistics," Naval Research Logistics Quarterly, John Wiley & Sons, vol. 1(2), pages 141-153, June.
    7. David Silver & Julian Schrittwieser & Karen Simonyan & Ioannis Antonoglou & Aja Huang & Arthur Guez & Thomas Hubert & Lucas Baker & Matthew Lai & Adrian Bolton & Yutian Chen & Timothy Lillicrap & Fan , 2017. "Mastering the game of Go without human knowledge," Nature, Nature, vol. 550(7676), pages 354-359, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lei Hai & Chenhao Zhou & Li Xue, 2026. "A Self-Adaptive Monte Carlo Tree Search Algorithm for Generalized Quay Crane Scheduling Problem," Transportation Science, INFORMS, vol. 60(1), pages 132-154, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Boute, Robert N. & Gijsbrechts, Joren & van Jaarsveld, Willem & Vanvuchelen, Nathalie, 2022. "Deep reinforcement learning for inventory control: A roadmap," European Journal of Operational Research, Elsevier, vol. 298(2), pages 401-412.
    2. Joren Gijsbrechts & Robert N. Boute & Jan A. Van Mieghem & Dennis J. Zhang, 2022. "Can Deep Reinforcement Learning Improve Inventory Management? Performance on Lost Sales, Dual-Sourcing, and Multi-Echelon Problems," Manufacturing & Service Operations Management, INFORMS, vol. 24(3), pages 1349-1368, May.
    3. Xiaoyue Li & John M. Mulvey, 2023. "Optimal Portfolio Execution in a Regime-switching Market with Non-linear Impact Costs: Combining Dynamic Program and Neural Network," Papers 2306.08809, arXiv.org.
    4. Mahmoud Mahfouz & Angelos Filos & Cyrine Chtourou & Joshua Lockhart & Samuel Assefa & Manuela Veloso & Danilo Mandic & Tucker Balch, 2019. "On the Importance of Opponent Modeling in Auction Markets," Papers 1911.12816, arXiv.org.
    5. Dawei Chen & Fangxu Mo & Ye Chen & Jun Zhang & Xinyu You, 2022. "Optimization of Ramp Locations along Freeways: A Dynamic Programming Approach," Sustainability, MDPI, vol. 14(15), pages 1-13, August.
    6. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    7. Yong Dam Jeong & Kwang Su Kim & Yunil Roh & Sooyoun Choi & Shingo Iwami & Il Hyo Jung, 2022. "Optimal Feedback Control of Cancer Chemotherapy Using Hamilton–Jacobi–Bellman Equation," Complexity, John Wiley & Sons, vol. 2022(1).
    8. Vanvuchelen, Nathalie & De Boeck, Kim & Boute, Robert N., 2024. "Cluster-based lateral transshipments for the Zambian health supply chain," European Journal of Operational Research, Elsevier, vol. 313(1), pages 373-386.
    9. Bartłomiej Kocot & Paweł Czarnul & Jerzy Proficz, 2023. "Energy-Aware Scheduling for High-Performance Computing Systems: A Survey," Energies, MDPI, vol. 16(2), pages 1-28, January.
    10. Wadi Khalid Anuar & Lai Soon Lee & Hsin-Vonn Seow & Stefan Pickl, 2021. "A Multi-Depot Vehicle Routing Problem with Stochastic Road Capacity and Reduced Two-Stage Stochastic Integer Linear Programming Models for Rollout Algorithm," Mathematics, MDPI, vol. 9(13), pages 1-44, July.
    11. Pulak Swain & Akshay Kumar Ojha, 2025. "Feasibility conditions of robust portfolio solutions with single and combined uncertainties," OPSEARCH, Springer;Operational Research Society of India, vol. 62(3), pages 1262-1287, September.
    12. Peter Schober & Julian Valentin & Dirk Pflüger, 2022. "Solving High-Dimensional Dynamic Portfolio Choice Models with Hierarchical B-Splines on Sparse Grids," Computational Economics, Springer;Society for Computational Economics, vol. 59(1), pages 185-224, January.
    13. Pavirani, Fabio & Van Gompel, Jonas & Karimi Madahi, Seyed Soroush & Claessens, Bert & Develder, Chris, 2025. "Predicting and publishing accurate imbalance prices using Monte Carlo Tree Search," Applied Energy, Elsevier, vol. 392(C).
    14. Matthias Breuer & David Windisch, 2019. "Investment Dynamics and Earnings‐Return Properties: A Structural Approach," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 57(3), pages 639-674, June.
    15. Diefenbach, Heiko & Emde, Simon & Glock, Christoph H., 2020. "Loading tow trains ergonomically for just-in-time part supply," European Journal of Operational Research, Elsevier, vol. 284(1), pages 325-344.
    16. Michael J. Pennock & William B. Rouse & Diane L. Kollar, 2007. "Transforming the Acquisition Enterprise: A Framework for Analysis and a Case Study of Ship Acquisition," Systems Engineering, John Wiley & Sons, vol. 10(2), pages 99-117, June.
    17. Quetschlich, Mathias & Moetz, André & Otto, Boris, 2021. "Optimisation model for multi-item multi-echelon supply chains with nested multi-level products," European Journal of Operational Research, Elsevier, vol. 290(1), pages 144-158.
    18. Iman Ahmadianfar & Saeed Noshadian & Nadir Ahmed Elagib & Meysam Salarijazi, 2021. "Robust Diversity-based Sine-Cosine Algorithm for Optimizing Hydropower Multi-reservoir Systems," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 35(11), pages 3513-3538, September.
    19. Marwan Shams Eddin & Hadi El-Amine, 2025. "Optimal Incentive-based Plans to Reduce Absenteeism," Management and Labour Studies, XLRI Jamshedpur, School of Business Management & Human Resources, vol. 50(3), pages 306-319, August.
    20. Zhenchong Mo & Lin Gong & Mingren Zhu & Junde Lan, 2024. "The Generative Generic-Field Design Method Based on Design Cognition and Knowledge Reasoning," Sustainability, MDPI, vol. 16(22), pages 1-34, November.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orijoc:v:37:y:2025:i:4:p:785-807. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.