IDEAS home Printed from https://ideas.repec.org/a/eee/proeco/v296y2026ics0925527326000708.html

Digital twin-enabled nested Q-learning for multi-layer production planning and inventory control

Author

Listed:
  • Liang, Erickson
  • Chang, Kuo-Hao

Abstract

Modern production systems face increasing uncertainty due to demand variability, supply delays, and machine breakdowns. Traditional hierarchical Production Planning and Inventory Control (PPIC) approaches often lack the adaptability required to manage such volatility across multiple planning horizons. This study proposes a digital twin-enabled nested Q-learning framework for coordinating strategic and operational decisions across monthly, weekly, and daily levels in a flexible job shop environment. The digital twin acts as a real-time simulation layer, capturing dynamic system states such as inventory levels, machine availability, and disruption events. A hierarchical reinforcement learning (HRL) structure, based on nested Q-learning agents, is used to learn adaptive policies for production planning, material procurement, and job scheduling. Computational experiments under stochastic conditions show that the proposed approach achieves cost reductions of 4%–15%, fill rates up to 99%, and improved machine utilization, outperforming conventional rule-based strategies. By integrating real-time feedback with multi-layer decision-making, this framework contributes to the development of autonomous, resilient, and Industry 4.0–ready production systems.

Suggested Citation

  • Liang, Erickson & Chang, Kuo-Hao, 2026. "Digital twin-enabled nested Q-learning for multi-layer production planning and inventory control," International Journal of Production Economics, Elsevier, vol. 296(C).
  • Handle: RePEc:eee:proeco:v:296:y:2026:i:c:s0925527326000708
    DOI: 10.1016/j.ijpe.2026.109979
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0925527326000708
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ijpe.2026.109979?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Soman, C.A. & van Donk, D.P. & Gaalman, G.J.C., 2007. "Capacitated planning and scheduling for combined make-to-order and make-to-stock production in the food industry: An illustrative case study," International Journal of Production Economics, Elsevier, vol. 108(1-2), pages 191-199, July.
    2. Mustapha Sali & Yahya Ghrab & Clément Chatras, 2023. "Optimal product aggregation for sales and operations planning in mass customisation context," Post-Print hal-04319422, HAL.
    3. Sali, Mustapha & Ghrab, Yahya & Chatras, Clément, 2023. "Optimal product aggregation for sales and operations planning in mass customisation context," International Journal of Production Economics, Elsevier, vol. 263(C).
    4. Rakiz, Asma & Absi, Nabil & Fenies, Pierre, 2023. "Comparing approaches for a multi-level planning problem in a mining industry," International Journal of Production Economics, Elsevier, vol. 265(C).
    5. Tsan‐Ming Choi & Stein W. Wallace & Yulan Wang, 2018. "Big Data Analytics in Operations Management," Production and Operations Management, Production and Operations Management Society, vol. 27(10), pages 1868-1883, October.
    6. Pan, Yanghua & Zhong, Ray Y. & Qu, Ting & Ding, Liqiang & Zhang, Jun, 2024. "Multi-level digital twin-driven kitting-synchronized optimization for production logistics system," International Journal of Production Economics, Elsevier, vol. 271(C).
    7. Lim, Kendrik Yan Hong & Dang, Le Van & Chen, Chun-Hsien, 2024. "Incorporating supply and production digital twins to mitigate demand disruptions in multi-echelon networks," International Journal of Production Economics, Elsevier, vol. 273(C).
    8. Xiao, Jianhua & Ma, Siyuan & Wang, Shuyi & Huang, George Q., 2024. "Fine-grained digital twin sharing framework for smart construction through an incentive mechanism," International Journal of Production Economics, Elsevier, vol. 276(C).
    9. Gabriel R. Bitran & Elizabeth A. Haas & Arnoldo C. Hax, 1981. "Hierarchical Production Planning: A Single Stage System," Operations Research, INFORMS, vol. 29(4), pages 717-743, August.
    10. Zhang, Mengdi & Yang, Wanting & Zhao, Zhiheng & Wang, Shuaian & Huang, George Q., 2024. "Do fairness concerns matter for ESG decision-making? Strategic interactions in digital twin-enabled sustainable semiconductor supply chain," International Journal of Production Economics, Elsevier, vol. 276(C).
    11. Aldrighetti, Riccardo & Battini, Daria & Ivanov, Dmitry & Zennaro, Ilenia, 2021. "Costs of resilience and disruptions in supply chain network design models: A review and future research directions," International Journal of Production Economics, Elsevier, vol. 235(C).
    12. Nguyen, Tiep & Duong, Quang Huy & Nguyen, Truong Van & Zhu, You & Zhou, Li, 2022. "Knowledge mapping of digital twin and physical internet in Supply Chain Management: A systematic literature review," International Journal of Production Economics, Elsevier, vol. 244(C).
    13. Hosseini-Motlagh, Seyyed-Mahdi & Samani, Mohammad Reza Ghatreh & Rahmani, Milad, 2025. "A digital twin framework integrated with a mixed proactive-reactive model for human milk supply chain planning," International Journal of Production Economics, Elsevier, vol. 287(C).
    14. Neto, Anis Assad & Ribeiro da Silva, Elias & Deschamps, Fernando & do Nascimento Junior, Laercio Alves & Pinheiro de Lima, Edson, 2023. "Modeling production disorder: Procedures for digital twins of flexibility-driven manufacturing systems," International Journal of Production Economics, Elsevier, vol. 260(C).
    15. Weidong Lei & Liu Yang & Pengyu Yan & Chengbin Chu & Jie Yang, 2025. "Production coordination of local and cloud orders in shared manufacturing: a bi-objective pre-scheduling approach," Annals of Operations Research, Springer, vol. 345(1), pages 207-245, February.
    16. Leung, Eric K.H. & Lee, Carmen Kar Hang & Ouyang, Zhiyuan, 2022. "From traditional warehouses to Physical Internet hubs: A digital twin-based inbound synchronization framework for PI-order management," International Journal of Production Economics, Elsevier, vol. 244(C).
    17. Hrabec, Dušan & Hvattum, Lars Magnus & Hoff, Arild, 2022. "The value of integrated planning for production, inventory, and routing decisions: A systematic review and meta-analysis," International Journal of Production Economics, Elsevier, vol. 248(C).
    18. Lee, Jongsuk & Chua, Ping Chong & Liu, Bufan & Moon, Seung Ki & Lopez, Manuel, 2025. "A hybrid data-driven optimization and decision-making approach for a digital twin environment: Towards customizing production platforms," International Journal of Production Economics, Elsevier, vol. 279(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kauffman, Marcos Eduardo & Soares, Marcelo Negri & Long, Tengfei & Harris, Lara & Cugula, Jarbas Rodrigues Gomes & Mazato, Welington Junior Jorge & Zhang, Kathy, 2026. "Enhancing intellectual property identification and valuation in manufacturing through digital twins," International Journal of Production Economics, Elsevier, vol. 294(C).
    2. Mosayebi, Mohsen & Fathi, Michel & Hedayati, Mehrnaz Khalaj & Ivanov, Dmitry, 2024. "Time-to-Adapt (TTA)," International Journal of Production Economics, Elsevier, vol. 278(C).
    3. Xiao, Jianhua & Li, Junning & Ma, Siyuan & Wang, Shuyi & Huang, George Q., 2026. "A new solution for sustainable manufacturing: The dynamics of digital twins and carbon futures under emission reduction targets," International Journal of Production Economics, Elsevier, vol. 295(C).
    4. Liu, Baolong & Wang, Weilong, 2023. "Live commerce retailing with online influencers: Two business models," International Journal of Production Economics, Elsevier, vol. 255(C).
    5. Mayr, Kathrin & Heininger, Richard & Teller, Christoph & Stary, Christian, 2025. "Beware of the customer: Managing customer deviance in last mile logistics," International Journal of Production Economics, Elsevier, vol. 289(C).
    6. Sahoo, Saumyaranjan & Kumar, Ashwani & Mangla, Sachin Kumar & Emrouznejad, Ali, 2026. "Enhancing supply chain resilience through digital twin-enabled risk management: The role of logistic service coordination and distribution network efficiency," International Journal of Production Economics, Elsevier, vol. 297(C).
    7. Roberta Pellegrino & Barbara Gaudenzi, 2026. "Supply chain resilience as a viable way to cope with disruptions: an empirical analysis of Italian firms," Review of Managerial Science, Springer, vol. 20(5), pages 1459-1489, May.
    8. Maheshwari, Pratik & Kamble, Sachin & Belhadi, Amine & Venkatesh, Mani & Abedin, Mohammad Zoynul, 2023. "Digital twin-driven real-time planning, monitoring, and controlling in food supply chains," Technological Forecasting and Social Change, Elsevier, vol. 195(C).
    9. Ivanov, Dmitry, 2023. "Intelligent digital twin (iDT) for supply chain stress-testing, resilience, and viability," International Journal of Production Economics, Elsevier, vol. 263(C).
    10. João Rampon Neto & Guilherme Bergmann Borges Vieira & Gabriel Vidor & Virginia Fani & Romeo Bandinelli & Miguel Afonso Sellitto, 2025. "Integrating portfolio decisions into sales and operations planning: a literature review and novel framework for holistic decision-making," Review of Managerial Science, Springer, vol. 19(9), pages 2775-2832, September.
    11. Lim, Kendrik Yan Hong & Dang, Le Van & Chen, Chun-Hsien, 2024. "Incorporating supply and production digital twins to mitigate demand disruptions in multi-echelon networks," International Journal of Production Economics, Elsevier, vol. 273(C).
    12. Wu, Hang & Li, Ming & Yu, Chenglin & Ouyang, Zhiyuan & Lai, Kee-hung & Zhao, Zhiheng & Pan, Shenle & Wang, Shuaian & Zhong, Ray Y. & Kuo, Yong-Hong & Zhang, Fangni & Huang, Wenjie & Shen, Zuo-Jun Max , 2025. "Towards cyber-physical internet: A systematic review, fundamental model and future perspectives," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 197(C).
    13. Abhishek Behl & Brinda Sampat & Vijay Pereira & Charbel José Chiappetta Jabbour, 2025. "The role played by responsible artificial intelligence (RAI) in improving supply chain performance in the MSME sector: an empirical inquiry," Annals of Operations Research, Springer, vol. 354(1), pages 5-34, November.
    14. Emilia Vann Yaroson & Soumyadeb Chowdhury & Sachin Kumar Mangla & Prasanta Kumar Dey, 2024. "Unearthing the interplay between organisational resources, knowledge and industry 4.0 analytical decision support tools to achieve sustainability and supply chain wellbeing," Annals of Operations Research, Springer, vol. 342(2), pages 1321-1368, November.
    15. Liu, Weihua & George Shanthikumar, J. & Tae-Woo Lee, Paul & Li, Xiang & Zhou, Li, 2021. "Special issue editorial: Smart supply chains and intelligent logistics services," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 147(C).
    16. Murugaiyan, Pachayappan & Kundu, Tanmoy & Kapoor, Rohit & Rengasamy, Sundar, 2026. "Physical internet drone logistics model: Optimizing cost and energy efficiency for urban delivery systems," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 212(C).
    17. Boysen, Nils & Briskorn, Dirk & Montreuil, Benoit & Zey, Lennart, 2025. "The π-transportation problem: On the value of split transports for the Physical Internet concept," European Journal of Operational Research, Elsevier, vol. 324(2), pages 629-643.
    18. Matthias Klumpp & Dominic Loske, 2021. "Sustainability and Resilience Revisited: Impact of Information Technology Disruptions on Empirical Retail Logistics Efficiency," Sustainability, MDPI, vol. 13(10), pages 1-20, May.
    19. Jingxin Zhang & Jianxiong Zhang & Chunqiu Xu, 2024. "Contract design considering data driven marketing: with and without the cap and trade regulation," Annals of Operations Research, Springer, vol. 333(1), pages 157-199, February.
    20. Gahm, Christian & Uzunoglu, Aykut & Wahl, Stefan & Ganschinietz, Chantal & Tuma, Axel, 2022. "Applying machine learning for the anticipation of complex nesting solutions in hierarchical production planning," European Journal of Operational Research, Elsevier, vol. 296(3), pages 819-836.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:proeco:v:296:y:2026:i:c:s0925527326000708. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijpe .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.