IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v16y2025i1d10.1038_s41467-025-60085-w.html
   My bibliography  Save this article

Memristive Bellman solver for decision-making

Author

Listed:
  • Zhe Feng

    (Anhui University)

  • Zuheng Wu

    (Anhui University)

  • Jianxun Zou

    (Anhui University)

  • Lingli Cheng

    (Fudan University)

  • Xiaolong Zhao

    (University of Science and Technology of China)

  • Xumeng Zhang

    (Fudan University)

  • Jian Lu

    (Zhejiang Laboratory)

  • Cong Wang

    (Nanjing University)

  • Yilin Wang

    (University of Science and Technology of China)

  • Haochen Wang

    (Anhui University)

  • Wenbin Guo

    (Anhui University)

  • Zhibin Qian

    (Anhui University)

  • Yunlai Zhu

    (Anhui University)

  • Zuyu Xu

    (Anhui University)

  • Yuehua Dai

    (Anhui University)

  • Qi Liu

    (Fudan University)

Abstract

The Bellman equation, with a resource-consuming solving process, plays a fundamental role in formulating and solving dynamic optimization problems. The realization of the Bellman solver with memristive computing-in-memory (MCIM) technology, is significant for implementing efficient dynamic decision-making. However, the iterative nature of the Bellman equation solving process poses a challenge for efficient implementation on MCIM systems, which excel at vector-matrix multiplication (VMM) operations but are less suited for iterative algorithms. In this work, by incorporating the temporal dimension and transforming the solution into recurrent dot product operations, a memristive Bellman solver (MBS) is proposed, facilitating the implementation of the Bellman equation solving process with efficient MCIM technology. The MBS effectively reduces the iteration numbers and which further enhanced by approximated solutions leveraging memristor noise. Finally, the path planning tasks are used to verify the feasibility of the proposed MBS. The theoretical derivation and experimental results demonstrate that the MBS effectively reduces the iteration cycles, facilitating the solving efficiency. This work could be a sound of choice for developing high-efficiency decision-making systems.

Suggested Citation

  • Zhe Feng & Zuheng Wu & Jianxun Zou & Lingli Cheng & Xiaolong Zhao & Xumeng Zhang & Jian Lu & Cong Wang & Yilin Wang & Haochen Wang & Wenbin Guo & Zhibin Qian & Yunlai Zhu & Zuyu Xu & Yuehua Dai & Qi L, 2025. "Memristive Bellman solver for decision-making," Nature Communications, Nature, vol. 16(1), pages 1-11, December.
    • Handle: RePEc:nat:natcom:v:16:y:2025:i:1:d:10.1038_s41467-025-60085-w
    DOI: 10.1038/s41467-025-60085-w
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-025-60085-w
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-025-60085-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Vinay Joshi & Manuel Le Gallo & Simon Haefeli & Irem Boybat & S. R. Nandakumar & Christophe Piveteau & Martino Dazzi & Bipin Rajendran & Abu Sebastian & Evangelos Eleftheriou, 2020. "Accurate deep neural network inference using computational phase-change memory," Nature Communications, Nature, vol. 11(1), pages 1-13, December.
    2. Irem Boybat & Manuel Le Gallo & S. R. Nandakumar & Timoleon Moraitis & Thomas Parnell & Tomas Tuma & Bipin Rajendran & Yusuf Leblebici & Abu Sebastian & Evangelos Eleftheriou, 2018. "Neuromorphic computing with multi-memristive synapses," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
    3. M. R. Mahmoodi & M. Prezioso & D. B. Strukov, 2019. "Versatile stochastic dot product circuits based on nonvolatile memories for high performance neurocomputing and neurooptimization," Nature Communications, Nature, vol. 10(1), pages 1-10, December.
    4. Mingyi Rao & Hao Tang & Jiangbin Wu & Wenhao Song & Max Zhang & Wenbo Yin & Ye Zhuo & Fatemeh Kiani & Benjamin Chen & Xiangqi Jiang & Hefei Liu & Hung-Yu Chen & Rivu Midya & Fan Ye & Hao Jiang & Zhong, 2023. "Thousands of conductance levels in memristors integrated on CMOS," Nature, Nature, vol. 615(7954), pages 823-829, March.
    5. Dong-Hyeok Lim & Shuang Wu & Rong Zhao & Jung-Hoon Lee & Hongsik Jeong & Luping Shi, 2021. "Spontaneous sparse learning for PCM-based memristor neural networks," Nature Communications, Nature, vol. 12(1), pages 1-14, December.
    6. John Rust, 1997. "Using Randomization to Break the Curse of Dimensionality," Econometrica, Econometric Society, vol. 65(3), pages 487-516, May.
    7. Ik-Jyae Kim & Min-Kyu Kim & Jang-Sik Lee, 2023. "Highly-scaled and fully-integrated 3-dimensional ferroelectric transistor array for hardware implementation of neural networks," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    8. Ruoqing Zhu & Ying-Qi Zhao & Guanhua Chen & Shuangge Ma & Hongyu Zhao, 2017. "Greedy outcome weighted tree learning of optimal personalized treatment rules," Biometrics, The International Biometric Society, vol. 73(2), pages 391-400, June.
    9. Peng Yao & Huaqiang Wu & Bin Gao & Sukru Burc Eryilmaz & Xueyao Huang & Wenqiang Zhang & Qingtian Zhang & Ning Deng & Luping Shi & H.-S. Philip Wong & He Qian, 2017. "Face classification using electronic synapses," Nature Communications, Nature, vol. 8(1), pages 1-8, August.
    10. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    11. L. G. Mitten, 1974. "Preference Order Dynamic Programming," Management Science, INFORMS, vol. 21(1), pages 43-46, September.
    12. Ádám Papp & Wolfgang Porod & Gyorgy Csaba, 2021. "Nanoscale neural network using non-linear spin-wave interference," Nature Communications, Nature, vol. 12(1), pages 1-8, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhiyuan Li & Zhongshao Li & Wei Tang & Jiaping Yao & Zhipeng Dou & Junjie Gong & Yongfei Li & Beining Zhang & Yunxiao Dong & Jian Xia & Lin Sun & Peng Jiang & Xun Cao & Rui Yang & Xiangshui Miao & Ron, 2024. "Crossmodal sensory neurons based on high-performance flexible memristors for human-machine in-sensor computing system," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    2. Peng Chen & Fenghao Liu & Peng Lin & Peihong Li & Yu Xiao & Bihua Zhang & Gang Pan, 2023. "Open-loop analog programmable electrochemical memory array," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    3. Rohit Abraham John & Yiğit Demirağ & Yevhen Shynkarenko & Yuliia Berezovska & Natacha Ohannessian & Melika Payvand & Peng Zeng & Maryna I. Bodnarchuk & Frank Krumeich & Gökhan Kara & Ivan Shorubalko &, 2022. "Reconfigurable halide perovskite nanocrystal memristors for neuromorphic computing," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    4. Shi, Chengchun & Luo, Shikai & Le, Yuan & Zhu, Hongtu & Song, Rui, 2022. "Statistically efficient advantage learning for offline reinforcement learning in infinite horizons," LSE Research Online Documents on Economics 115598, London School of Economics and Political Science, LSE Library.
    5. Mingrui Jiang & Keyi Shan & Chengping He & Can Li, 2023. "Efficient combinatorial optimization by quantum-inspired parallel annealing in analogue memristor crossbar," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    6. Tulika Saha & Sriparna Saha & Pushpak Bhattacharyya, 2020. "Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-28, July.
    7. Nikolaj Malchow-Møller & Michael Svarer, 2003. "Estimation of the multinomial logit model with random effects," Applied Economics Letters, Taylor & Francis Journals, vol. 10(7), pages 389-392.
    8. Mahmoud Mahfouz & Angelos Filos & Cyrine Chtourou & Joshua Lockhart & Samuel Assefa & Manuela Veloso & Danilo Mandic & Tucker Balch, 2019. "On the Importance of Opponent Modeling in Auction Markets," Papers 1911.12816, arXiv.org.
    9. Lixiang Zhang & Yan Yan & Yaoguang Hu, 2024. "Deep reinforcement learning for dynamic scheduling of energy-efficient automated guided vehicles," Journal of Intelligent Manufacturing, Springer, vol. 35(8), pages 3875-3888, December.
    10. Benjamin Heinbach & Peter Burggräf & Johannes Wagner, 2024. "gym-flp: A Python Package for Training Reinforcement Learning Algorithms on Facility Layout Problems," SN Operations Research Forum, Springer, vol. 5(1), pages 1-26, March.
    11. Jiarui Han & Tze Lai & Viktor Spivakovsky, 2006. "Approximate Policy Optimization and Adaptive Control in Regression Models," Computational Economics, Springer;Society for Computational Economics, vol. 27(4), pages 433-452, June.
    12. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.
    13. Ertian Chen, 2025. "Model-Adaptive Approach to Dynamic Discrete Choice Models with Large State Spaces," Papers 2501.18746, arXiv.org, revised Jun 2025.
    14. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    15. Yuhong Wang & Lei Chen & Hong Zhou & Xu Zhou & Zongsheng Zheng & Qi Zeng & Li Jiang & Liang Lu, 2021. "Flexible Transmission Network Expansion Planning Based on DQN Algorithm," Energies, MDPI, vol. 14(7), pages 1-21, April.
    16. Oleksii M. Volkov & Oleksandr V. Pylypovskyi & Fabrizio Porrati & Florian Kronast & Jose A. Fernandez-Roldan & Attila Kákay & Alexander Kuprava & Sven Barth & Filipp N. Rybakov & Olle Eriksson & Sebas, 2024. "Three-dimensional magnetic nanotextures with high-order vorticity in soft magnetic wireframes," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    17. Pedro Reis & Ana Paula Serra & Jo~ao Gama, 2025. "The Role of Deep Learning in Financial Asset Management: A Systematic Review," Papers 2503.01591, arXiv.org.
    18. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    19. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    20. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:16:y:2025:i:1:d:10.1038_s41467-025-60085-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.