IDEAS home Printed from https://ideas.repec.org/a/spr/jcomop/v31y2016i3d10.1007_s10878-014-9820-3.html
   My bibliography  Save this article

A reinforcement-learning approach for admission control in distributed network service systems

Author

Listed:
  • Xiaonong Lu

    (University of Science and Technology of China)

  • Baoqun Yin

    (University of Science and Technology of China)

  • Haipeng Zhang

    (University of Science and Technology of China)

Abstract

In the distributed network service systems such as streaming-media systems and resource-sharing systems with multiple service nodes, admission control (AC) technology is an essential way to enhance performance. Model-based optimization approaches are good ways to be applied to analyze and solve the optimal AC policy. However, due to “the curse of dimensionality”, computing such policy for practical systems is a rather difficult task. In this paper, we consider a general model of the distributed network service systems, and address the problem of designing an optimal AC policy. An analytical model is presented for the system with fixed parameters based on semi-Markov decision process (SMDP). We design an event-driven AC policy, and the stationary randomized policy is taken as the policy structure. To solve the SMDP, both the state aggregation approach and the reinforcement-learning (RL) method with online policy optimization algorithm are applied. Then, we extend the problem by considering the system with time-varying parameters, where the arrival rates of requests at each service node may change over time. In view of this situation, an AC policy switching mechanism is presented. This mechanism allows the system to decide whether to adjust its AC policy according to the policy switching rule. And in order to maximize the gain of system, that is, to obtain the optimal AC policy switching rule, another RL-based algorithm is applied. To assess the effectiveness of SMDP-based AC policy and policy switching mechanism for the system, numerical experiments are presented. We compare the performance of optimal policies obtained by the solutions of proposed methods with other classical AC policies. The simulation results illustrate that higher performance and computational efficiency could be achieved by using the SMDP model and RL-based algorithms proposed in this paper.

Suggested Citation

  • Xiaonong Lu & Baoqun Yin & Haipeng Zhang, 2016. "A reinforcement-learning approach for admission control in distributed network service systems," Journal of Combinatorial Optimization, Springer, vol. 31(3), pages 1241-1268, April.
    • Handle: RePEc:spr:jcomop:v:31:y:2016:i:3:d:10.1007_s10878-014-9820-3
    DOI: 10.1007/s10878-014-9820-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10878-014-9820-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10878-014-9820-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gosavi, Abhijit, 2004. "Reinforcement learning for long-run average cost," European Journal of Operational Research, Elsevier, vol. 155(3), pages 654-674, June.
    2. Li, Yanjie & Cao, Fang, 2013. "A basic formula for performance gradient estimation of semi-Markov decision processes," European Journal of Operational Research, Elsevier, vol. 224(2), pages 333-339.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Duraikannan Sundaramoorthi & Victoria Chen & Jay Rosenberger & Seoung Kim & Deborah Buckley-Behan, 2010. "A data-integrated simulation-based optimization for assigning nurses to patient admissions," Health Care Management Science, Springer, vol. 13(3), pages 210-221, September.
    2. Li, Xueping & Wang, Jiao & Sawhney, Rapinder, 2012. "Reinforcement learning for joint pricing, lead-time and scheduling decisions in make-to-order systems," European Journal of Operational Research, Elsevier, vol. 221(1), pages 99-109.
    3. Singh, Sumeetpal S. & Tadic, Vladislav B. & Doucet, Arnaud, 2007. "A policy gradient method for semi-Markov decision processes with application to call admission control," European Journal of Operational Research, Elsevier, vol. 178(3), pages 808-818, May.
    4. Barlow, E. & Bedford, T. & Revie, M. & Tan, J. & Walls, L., 2021. "A performance-centred approach to optimising maintenance of complex systems," European Journal of Operational Research, Elsevier, vol. 292(2), pages 579-595.
    5. Safaei, Fatemeh & Ahmadi, Jafar & Taghipour, Sharareh, 2022. "A maintenance policy for a k-out-of-n system under enhancing the system’s operating time and safety constraints, and selling the second-hand components," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    6. Stephane R. A. Barde & Soumaya Yacout & Hayong Shin, 2019. "Optimal preventive maintenance policy based on reinforcement learning of a fleet of military trucks," Journal of Intelligent Manufacturing, Springer, vol. 30(1), pages 147-161, January.
    7. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    8. Tang, Hao & Xu, Lingling & Sun, Jing & Chen, Yingjun & Zhou, Lei, 2015. "Modeling and optimization control of a demand-driven, conveyor-serviced production station," European Journal of Operational Research, Elsevier, vol. 243(3), pages 839-851.
    9. Xia, Li & Shihada, Basem, 2015. "A Jackson network model and threshold policy for joint optimization of energy and delay in multi-hop wireless networks," European Journal of Operational Research, Elsevier, vol. 242(3), pages 778-787.
    10. Yang, Hongbing & Li, Wenchao & Wang, Bin, 2021. "Joint optimization of preventive maintenance and production scheduling for multi-state production systems based on reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    11. Peter Seele & Claus Dierksmeier & Reto Hofstetter & Mario D. Schultz, 2021. "Mapping the Ethicality of Algorithmic Pricing: A Review of Dynamic and Personalized Pricing," Journal of Business Ethics, Springer, vol. 170(4), pages 697-719, May.
    12. Schütz, Hans-Jörg & Kolisch, Rainer, 2012. "Approximate dynamic programming for capacity allocation in the service industry," European Journal of Operational Research, Elsevier, vol. 218(1), pages 239-250.
    13. Jian Wang & Murtaza Das & Stephen Tappert, 2021. "Applying reinforcement learning to estimating apartment reference rents," Journal of Revenue and Pricing Management, Palgrave Macmillan, vol. 20(3), pages 330-343, June.
    14. van Wezel, M.C. & van Eck, N.J.P., 2005. "Reinforcement learning and its application to Othello," Econometric Institute Research Papers EI 2005-47, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcomop:v:31:y:2016:i:3:d:10.1007_s10878-014-9820-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.