Deep reinforcement learning for joint optimization of maintenance and spare parts ordering considering spare parts supply uncertainty

Efficient maintenance and spare parts ordering strategies can reduce costs for manufacturing companies. In recent years, important components may suffer supply risks due to geopolitical conflicts, trade conflicts, and limitations of key resources. This paper investigates the joint optimization of condition-based maintenance and dual sourcing of spare parts from reliable and unreliable suppliers. We formulate this joint decision problem with a Markov decision process and design a value iteration algorithm to obtain exact solutions for the optimal maintenance and ordering policy. However, the value iteration algorithm is not suitable for solving large-scale problems due to its long running time. Thus, we develop a deep Q-network (DQN) algorithm based on deep reinforcement learning to improve computation efficiency. Numerical experiments are conducted to validate the effectiveness of the DQN algorithm. The results show that the DQN algorithm can reduce the running time by 92.58 % for systems with more than 4 components and more than 5 states within a 4.82 % cost gap compared to the value iteration algorithm. Compared to the separate heuristic policy, the DQN algorithm can averagely reduce the cost by 11.27 %.