Digital twin-enabled nested Q-learning for multi-layer production planning and inventory control

Modern production systems face increasing uncertainty due to demand variability, supply delays, and machine breakdowns. Traditional hierarchical Production Planning and Inventory Control (PPIC) approaches often lack the adaptability required to manage such volatility across multiple planning horizons. This study proposes a digital twin-enabled nested Q-learning framework for coordinating strategic and operational decisions across monthly, weekly, and daily levels in a flexible job shop environment. The digital twin acts as a real-time simulation layer, capturing dynamic system states such as inventory levels, machine availability, and disruption events. A hierarchical reinforcement learning (HRL) structure, based on nested Q-learning agents, is used to learn adaptive policies for production planning, material procurement, and job scheduling. Computational experiments under stochastic conditions show that the proposed approach achieves cost reductions of 4%–15%, fill rates up to 99%, and improved machine utilization, outperforming conventional rule-based strategies. By integrating real-time feedback with multi-layer decision-making, this framework contributes to the development of autonomous, resilient, and Industry 4.0–ready production systems.