IDEAS home Printed from https://ideas.repec.org/a/eee/proeco/v284y2025ics0925527325000866.html
   My bibliography  Save this article

Scalable deep reinforcement learning in the non-stationary capacitated lot sizing problem

Author

Listed:
  • van Hezewijk, Lotte
  • Dellaert, Nico P.
  • van Jaarsveld, Willem L.

Abstract

Capacitated lot sizing problems in situations with stationary and non-stationary demand (SCLSP) are very common in practice. Solving problems with a large number of items using Deep Reinforcement Learning (DRL) is challenging due to the large action space. This paper proposes a new Markov Decision Process (MDP) formulation to solve this problem, by decomposing the production quantity decisions in a period into sub-decisions, which reduces the action space dramatically. We demonstrate that applying Deep Controlled Learning (DCL) yields policies that outperform the benchmark heuristic as well as a prior DRL implementation. By using the decomposed MDP formulation and DCL method outlined in this paper, we can solve larger problems compared to the previous DRL implementation. Moreover, we adopt a non-stationary demand model for training the policy, which enables us to readily apply the trained policy in dynamic environments when demand changes.

Suggested Citation

  • van Hezewijk, Lotte & Dellaert, Nico P. & van Jaarsveld, Willem L., 2025. "Scalable deep reinforcement learning in the non-stationary capacitated lot sizing problem," International Journal of Production Economics, Elsevier, vol. 284(C).
  • Handle: RePEc:eee:proeco:v:284:y:2025:i:c:s0925527325000866
    DOI: 10.1016/j.ijpe.2025.109601
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0925527325000866
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ijpe.2025.109601?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Stranieri, Francesco & Fadda, Edoardo & Stella, Fabio, 2024. "Combining deep reinforcement learning and multi-stage stochastic programming to address the supply chain inventory management problem," International Journal of Production Economics, Elsevier, vol. 268(C).
    2. Hyndman, Rob J. & Koehler, Anne B. & Snyder, Ralph D. & Grose, Simone, 2002. "A state space framework for automatic forecasting using exponential smoothing methods," International Journal of Forecasting, Elsevier, vol. 18(3), pages 439-454.
    3. Babai, M. Zied & Dai, Yong & Li, Qinyun & Syntetos, Aris & Wang, Xun, 2022. "Forecasting of lead-time demand variance: Implications for safety stock calculations," European Journal of Operational Research, Elsevier, vol. 296(3), pages 846-861.
    4. Ma, Xiyuan & Rossi, Roberto & Archibald, Thomas Welsh, 2022. "Approximations for non-stationary stochastic lot-sizing under (s,Q)-type policy," European Journal of Operational Research, Elsevier, vol. 298(2), pages 573-584.
    5. Dehaybe, Henri & Catanzaro, Daniele & Chevalier, Philippe, 2024. "Deep Reinforcement Learning for inventory optimization with non-stationary uncertain demand," European Journal of Operational Research, Elsevier, vol. 314(2), pages 433-445.
    6. Tunc, Huseyin & Kilic, Onur A. & Tarim, S. Armagan & Eksioglu, Burak, 2011. "The cost of using stationary inventory policies when demand is non-stationary," Omega, Elsevier, vol. 39(4), pages 410-415, August.
    7. Prak, Dennis & Teunter, Ruud, 2019. "A general method for addressing forecasting uncertainty in inventory models," International Journal of Forecasting, Elsevier, vol. 35(1), pages 224-238.
    8. Erhan Bayraktar & Michael Ludkovski, 2010. "Inventory management with partially observed nonstationary demand," Annals of Operations Research, Springer, vol. 176(1), pages 7-39, April.
    9. Kaynov, Illya & van Knippenberg, Marijn & Menkovski, Vlado & van Breemen, Albert & van Jaarsveld, Willem, 2024. "Deep Reinforcement Learning for One-Warehouse Multi-Retailer inventory management," International Journal of Production Economics, Elsevier, vol. 267(C).
    10. Jing-Sheng Song & Paul Zipkin, 1993. "Inventory Control in a Fluctuating Demand Environment," Operations Research, INFORMS, vol. 41(2), pages 351-370, April.
    11. Stephen C. Graves, 1999. "A Single-Item Inventory Model for a Nonstationary Demand Process," Manufacturing & Service Operations Management, INFORMS, vol. 1(1), pages 50-61.
    12. Verleijsdonk, Peter & van Jaarsveld, Willem & Kapodistria, Stella, 2024. "Scalable policies for the dynamic traveling multi-maintainer problem with alerts," European Journal of Operational Research, Elsevier, vol. 319(1), pages 121-134.
    13. Jianqiang Hu & Cheng Zhang & Chenbo Zhu, 2016. "( s , S ) Inventory Systems with Correlated Demands," INFORMS Journal on Computing, INFORMS, vol. 28(4), pages 603-611, November.
    14. Stephen C. Graves, 1999. "Addendum to "A Single-Item Inventory Model for a Nonstationary Demand Process"," Manufacturing & Service Operations Management, INFORMS, vol. 1(2), pages 174-174.
    15. Strijbosch, Leo W.G. & Syntetos, Aris A. & Boylan, John E. & Janssen, Elleke, 2011. "On the interaction between forecasting and stock control: The case of non-stationary demand," International Journal of Production Economics, Elsevier, vol. 133(1), pages 470-480, September.
    16. Lotte van Hezewijk & Nico Dellaert & Tom Van Woensel & Noud Gademann, 2023. "Using the proximal policy optimisation algorithm for solving the stochastic capacitated lot sizing problem," International Journal of Production Research, Taylor & Francis Journals, vol. 61(6), pages 1955-1978, March.
    17. Joren Gijsbrechts & Robert N. Boute & Jan A. Van Mieghem & Dennis J. Zhang, 2022. "Can Deep Reinforcement Learning Improve Inventory Management? Performance on Lost Sales, Dual-Sourcing, and Multi-Echelon Problems," Manufacturing & Service Operations Management, INFORMS, vol. 24(3), pages 1349-1368, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lotte Hezewijk & Nico P. Dellaert & Willem L. Jaarsveld, 2025. "On non-negative auto-correlated integer demand processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 101(2), pages 135-161, April.
    2. Xiang, Mengyuan & Rossi, Roberto & Martin-Barragan, Belen & Tarim, S. Armagan, 2023. "A mathematical programming-based solution method for the nonstationary inventory problem under correlated demand," European Journal of Operational Research, Elsevier, vol. 304(2), pages 515-524.
    3. Xiang, Mengyuan & Rossi, Roberto & Martin-Barragan, Belen & Tarim, S. Armagan, 2018. "Computing non-stationary (s, S) policies using mixed integer linear programming," European Journal of Operational Research, Elsevier, vol. 271(2), pages 490-500.
    4. Ren, Ke & Bidkhori, Hoda & Shen, Zuo-Jun Max, 2024. "Data-driven inventory policy: Learning from sequentially observed non-stationary data," Omega, Elsevier, vol. 123(C).
    5. Yee, Hannah & van Staden, Heletjé E. & Boute, Robert N., 2024. "Dual sourcing under non-stationary demand and partial observability," European Journal of Operational Research, Elsevier, vol. 314(1), pages 94-110.
    6. Stößlein, Martin & Kanet, John Jack & Gorman, Mike & Minner, Stefan, 2014. "Time-phased safety stocks planning and its financial impacts: Empirical evidence based on European econometric data," International Journal of Production Economics, Elsevier, vol. 149(C), pages 47-55.
    7. Temizöz, Tarkan & Imdahl, Christina & Dijkman, Remco & Lamghari-Idrissi, Douniel & van Jaarsveld, Willem, 2025. "Deep Controlled Learning for Inventory Control," European Journal of Operational Research, Elsevier, vol. 324(1), pages 104-117.
    8. Saoud, Patrick & Kourentzes, Nikolaos & Boylan, John E., 2022. "Approximations for the Lead Time Variance: a Forecasting and Inventory Evaluation," Omega, Elsevier, vol. 110(C).
    9. Jianqiang Hu & Cheng Zhang & Chenbo Zhu, 2016. "( s , S ) Inventory Systems with Correlated Demands," INFORMS Journal on Computing, INFORMS, vol. 28(4), pages 603-611, November.
    10. Boxiao Chen, 2021. "Data‐Driven Inventory Control with Shifting Demand," Production and Operations Management, Production and Operations Management Society, vol. 30(5), pages 1365-1385, May.
    11. Layth C. Alwan & Christian H. Weiß, 2017. "INAR implementation of newsvendor model for serially dependent demand counts," International Journal of Production Research, Taylor & Francis Journals, vol. 55(4), pages 1085-1099, February.
    12. Yang, Liu & Li, Haitao & Campbell, James F. & Sweeney, Donald C., 2017. "Integrated multi-period dynamic inventory classification and control," International Journal of Production Economics, Elsevier, vol. 189(C), pages 86-96.
    13. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    14. Manafzadeh Dizbin, Nima & Tan, Barış, 2020. "Optimal control of production-inventory systems with correlated demand inter-arrival and processing times," International Journal of Production Economics, Elsevier, vol. 228(C).
    15. Srinagesh Gavirneni & Sridhar Tayur, 1999. "Managing a Customer Following a Target Reverting Policy," Manufacturing & Service Operations Management, INFORMS, vol. 1(2), pages 157-173.
    16. Gen Sakoda & Hideki Takayasu & Misako Takayasu, 2019. "Data Science Solutions for Retail Strategy to Reduce Waste Keeping High Profit," Sustainability, MDPI, vol. 11(13), pages 1-30, June.
    17. Altay, Nezih & Litteral, Lewis A. & Rudisill, Frank, 2012. "Effects of correlation on intermittent demand forecasting and stock control," International Journal of Production Economics, Elsevier, vol. 135(1), pages 275-283.
    18. Lucy Gongtao Chen & Lawrence W. Robinson & Robin O. Roundy & Rachel Q. Zhang, 2015. "Technical Note—New Sufficient Conditions for ( s, S ) Policies to be Optimal in Systems with Multiple Uncertainties," Operations Research, INFORMS, vol. 63(1), pages 186-197, February.
    19. Fleuren, Tijn, 2025. "Stochastic approaches for production-inventory planning : Applications to high-tech supply chains," Other publications TiSEM 1fe1bbe5-fd90-4077-8606-d, Tilburg University, School of Economics and Management.
    20. Mirko Kremer & Brent Moritz & Enno Siemsen, 2011. "Demand Forecasting Behavior: System Neglect and Change Detection," Management Science, INFORMS, vol. 57(10), pages 1827-1843, October.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:proeco:v:284:y:2025:i:c:s0925527325000866. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijpe .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.