IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v347y2025i3d10.1007_s10479-025-06553-4.html
   My bibliography  Save this article

Markov decision process and approximate dynamic programming for a patient assignment scheduling problem

Author

Listed:
  • Małgorzata M. O’Reilly

    (University of Tasmania)

  • Sebastian Krasnicki

    (University of Tasmania)

  • James Montgomery

    (University of Tasmania)

  • Mojtaba Heydar

    (BHP)

  • Richard Turner

    (University of Tasmania)

  • Pieter Dam

    (University of Tasmania)

  • Peter Maree

    (Department of Health)

Abstract

We study the patient assignment scheduling (PAS) problem in a random environment that arises in the management of patient flow in hospital systems, due to the stochastic nature of the arrivals as well as the length of stay (LoS) distribution. At the start of each time period, emergency patients in the waiting area of a hospital system need to be admitted to relevant wards. Decisions may involve allocation to less suitable wards, or transfers of the existing inpatients to accommodate higher priority cases when wards are at full capacity. However, the LoS for patients in non-primary wards may increase, potentially leading to long-term congestion. To assist with decision-making in this PAS problem, we construct a discrete-time Markov decision process over an infinite horizon, with multiple patient types and multiple wards. Since the instances of realistic size of this problem are not easy to solve, we develop numerical methods based on approximate dynamic programming. We demonstrate the application potential of our methodology under practical considerations with numerical examples, using parameters obtained from data at a tertiary referral hospital in Australia. We gain valuable insights, such as the number of patients in non-primary wards, the number of transferred patients, and the number of patients redirected to other facilities, under different policies that enhance the system’s performance. This approach allows for more realistic assumptions and can also help determine the appropriate size of wards for different patient types within the hospital system.

Suggested Citation

  • Małgorzata M. O’Reilly & Sebastian Krasnicki & James Montgomery & Mojtaba Heydar & Richard Turner & Pieter Dam & Peter Maree, 2025. "Markov decision process and approximate dynamic programming for a patient assignment scheduling problem," Annals of Operations Research, Springer, vol. 347(3), pages 1493-1531, April.
    • Handle: RePEc:spr:annopr:v:347:y:2025:i:3:d:10.1007_s10479-025-06553-4
    DOI: 10.1007/s10479-025-06553-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-025-06553-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-025-06553-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. J. G. Dai & Pengyi Shi, 2019. "Inpatient Overflow: An Approximate Dynamic Programming Approach," Manufacturing & Service Operations Management, INFORMS, vol. 21(4), pages 894-911, October.
    2. Christiane Barz & Kumar Rajaram, 2015. "Elective Patient Admission and Scheduling under Multiple Resource Constraints," Production and Operations Management, Production and Operations Management Society, vol. 24(12), pages 1907-1930, December.
    3. Yasin Gocgun, 2018. "Simulation-based approximate policy iteration for dynamic patient scheduling for radiation therapy," Health Care Management Science, Springer, vol. 21(3), pages 317-325, September.
    4. Jie Bai & Andreas Fügener & Jochen Gönsch & Jens O. Brunner & Manfred Blobner, 2021. "Managing admission and discharge processes in intensive care units," Health Care Management Science, Springer, vol. 24(4), pages 666-685, December.
    5. Wim Vancroonenburg & Patrick Causmaecker & Greet Vanden Berghe, 2016. "A study of decision support models for online patient-to-room assignment planning," Annals of Operations Research, Springer, vol. 239(1), pages 253-271, April.
    6. Warren B. Powell, 2009. "What you should know about approximate dynamic programming," Naval Research Logistics (NRL), John Wiley & Sons, vol. 56(3), pages 239-249, April.
    7. Warren B. Powell, 2016. "Perspectives of approximate dynamic programming," Annals of Operations Research, Springer, vol. 241(1), pages 319-356, June.
    8. Saied Samiedaluie & Beste Kucukyazici & Vedat Verter & Dan Zhang, 2017. "Managing Patient Admissions in a Neurology Ward," Operations Research, INFORMS, vol. 65(3), pages 635-656, June.
    9. Zhang, Jian & Dridi, Mahjoub & El Moudni, Abdellah, 2019. "A two-level optimization model for elective surgery scheduling with downstream capacity constraints," European Journal of Operational Research, Elsevier, vol. 276(2), pages 602-613.
    10. Anders Reenberg Andersen & Thomas Jacob Riis Stidsen & Line Blander Reinhardt, 2020. "Simulation-Based Rolling Horizon Scheduling for Operating Theatres," SN Operations Research Forum, Springer, vol. 1(2), pages 1-26, June.
    11. Peter J. H. Hulshof & Martijn R. K. Mes & Richard J. Boucherie & Erwin W. Hans, 2016. "Patient admission planning using Approximate Dynamic Programming," Flexible Services and Manufacturing Journal, Springer, vol. 28(1), pages 30-61, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gökalp, E. & Gülpınar, N. & Doan, X.V., 2023. "Dynamic surgery management under uncertainty," European Journal of Operational Research, Elsevier, vol. 309(2), pages 832-844.
    2. Zamani, Hamed & Parvaresh, Fereshteh & Izady, Navid & Zanjirani Farahani, Reza, 2024. "Admission, discharge, and transfer control in patient flow logistics: Overview and future research," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 191(C).
    3. Mojtaba Heydar & Małgorzata M. O’Reilly & Erin Trainer & Mark Fackrell & Peter G. Taylor & Ali Tirdad, 2022. "A stochastic model for the patient-bed assignment problem with random arrivals and departures," Annals of Operations Research, Springer, vol. 315(2), pages 813-845, August.
    4. Liping Zhou & Na Geng & Zhibin Jiang & Shan Jiang, 2022. "Integrated Multiresource Capacity Planning and Multitype Patient Scheduling," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 129-149, January.
    5. Hessam Bavafa & Charles M. Leys & Lerzan Örmeci & Sergei Savin, 2019. "Managing Portfolio of Elective Surgical Procedures: A Multidimensional Inverse Newsvendor Problem," Operations Research, INFORMS, vol. 67(6), pages 1543-1563, November.
    6. Guido, Rosita & Groccia, Maria Carmela & Conforti, Domenico, 2018. "An efficient matheuristic for offline patient-to-bed assignment problems," European Journal of Operational Research, Elsevier, vol. 268(2), pages 486-503.
    7. Silva, Thiago A.O. & de Souza, Mauricio C., 2020. "Surgical scheduling under uncertainty by approximate dynamic programming," Omega, Elsevier, vol. 95(C).
    8. Carri W. Chan & Linda V. Green & Suparerk Lekwijit & Lijian Lu & Gabriel Escobar, 2019. "Assessing the Impact of Service Level When Customer Needs Are Uncertain: An Empirical Investigation of Hospital Step-Down Units," Management Science, INFORMS, vol. 65(2), pages 751-775, February.
    9. Jie Bai & Andreas Fügener & Jochen Gönsch & Jens O. Brunner & Manfred Blobner, 2021. "Managing admission and discharge processes in intensive care units," Health Care Management Science, Springer, vol. 24(4), pages 666-685, December.
    10. Ma, Xin & Zhao, Xue & Guo, Pengfei, 2022. "Cope with the COVID-19 pandemic: Dynamic bed allocation and patient subsidization in a public healthcare system," International Journal of Production Economics, Elsevier, vol. 243(C).
    11. Sean Harris & David Claudio, 2022. "Current Trends in Operating Room Scheduling 2015 to 2020: a Literature Review," SN Operations Research Forum, Springer, vol. 3(1), pages 1-42, March.
    12. Fabian Schäfer & Manuel Walther & Dominik G. Grimm & Alexander Hübner, 2023. "Combining machine learning and optimization for the operational patient-bed assignment problem," Health Care Management Science, Springer, vol. 26(4), pages 785-806, December.
    13. Majed Hadid & Adel Elomri & Tarek Mekkawy & Laoucine Kerbache & Abdelfatteh Omri & Halima Omri & Ruba Y. Taha & Anas Ahmad Hamad & Mohammed Hamad J. Thani, 2022. "Bibliometric analysis of cancer care operations management: current status, developments, and future directions," Health Care Management Science, Springer, vol. 25(1), pages 166-185, March.
    14. Aisha Tayyab & Saif Ullah & Mohammed Fazle Baki, 2023. "An Outer Approximation Method for Scheduling Elective Surgeries with Sequence Dependent Setup Times to Multiple Operating Rooms," Mathematics, MDPI, vol. 11(11), pages 1-15, May.
    15. Jian-Jun Wang & Zongli Dai & Ai-Chih Chang & Jim Junmin Shi, 2022. "Surgical scheduling by Fuzzy model considering inpatient beds shortage under uncertain surgery durations," Annals of Operations Research, Springer, vol. 315(1), pages 463-505, August.
    16. Heydar, Mojtaba & Mardaneh, Elham & Loxton, Ryan, 2022. "Approximate dynamic programming for an energy-efficient parallel machine scheduling problem," European Journal of Operational Research, Elsevier, vol. 302(1), pages 363-380.
    17. Dai, Jiajun & Geng, Na & Xie, Xiaolan, 2021. "Dynamic advance scheduling of outpatient appointments in a moving booking window," European Journal of Operational Research, Elsevier, vol. 292(2), pages 622-632.
    18. Minglong Zhou & Melvyn Sim & Shao‐Wei Lam, 2022. "Advance admission scheduling via resource satisficing," Production and Operations Management, Production and Operations Management Society, vol. 31(11), pages 4002-4020, November.
    19. Rempel, M. & Cai, J., 2021. "A review of approximate dynamic programming applications within military operations research," Operations Research Perspectives, Elsevier, vol. 8(C).
    20. Jim G. Dai & Pengyi Shi, 2021. "Recent Modeling and Analytical Advances in Hospital Inpatient Flow Management," Production and Operations Management, Production and Operations Management Society, vol. 30(6), pages 1838-1862, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:347:y:2025:i:3:d:10.1007_s10479-025-06553-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.