A multi-agent deep reinforcement learning approach for multi-echelon inventory optimization and its application to the beer game

Cost-effective inventory management in supply chain networks remains challenging since uncertainties may be distributed along with multi-echelon inventories. This paper investigates a decentralized multi-echelon inventory optimization problem with stochastic demands, focusing on a serial four-echelon supply chain in the beer game. In this problem, participants at each stage independently execute their replenishment policies based on local information, while collectively minimizing the total cost of the entire supply chain. We formulate this problem as a decentralized partially observable Markov decision process and develop a novel multi-agent deep reinforcement learning (MADRL) algorithm to address it, without relying on any stylized problem assumptions. Additionally, given that MADRL usually trains agents in simulated environments and the policy obtained may perform poorly in target environments due to environmental discrepancies, we propose a domain randomization technique integrated into MADRL, called DR-MADRL, to enhance the robustness of learned policies. Numerical experiments show that, when simulated and target environments align, MADRL’s policies come close to optimum and exhibit structural properties similar to the base-stock policy that is optimal analytically. The superiority of MADRL is further demonstrated as it surpasses existing approximate or heuristic methods in instances devoid of analytically optimal policies. Meanwhile, the agents demonstrate certain behavioral characteristics, which could aid in identifying or developing new replenishment policies. For scenarios with environmental discrepancies, DR-MADRL’s policy demonstrates substantial resilience to supply chain variations. These results show the effectiveness of MADRL and DR-MADRL in addressing the multi-echelon inventory optimization problem.