IDEAS home Printed from https://ideas.repec.org/a/eee/transb/v167y2023icp196-216.html
   My bibliography  Save this article

Hierarchical control for stochastic network traffic with reinforcement learning

Author

Listed:
  • Su, Z.C.
  • Chow, Andy H.F.
  • Fang, C.L.
  • Liang, E.M.
  • Zhong, R.X.

Abstract

This study proposes a hierarchical control framework to maximize the throughput of a road network driven by travel demand with uncertainties. In the upper level, a perimeter controller regulates the traffic influx into the core road network. The upper level uses a reinforcement learning algorithm that learns and responds to the traffic dynamics in the core road network without the need for an underlying system model and macroscopic fundamental diagram. The lower level is a local signal control system that regulates the spatial distribution of traffic flow within the core network. The results show that the hierarchical control framework can improve road network throughput by coordinating control actions conducted at the two levels. The improvement in system-wide performance is validated by a range of performance metrics and macroscopic flow-accumulation patterns realized under different control settings. The study contributes to the management of urban road networks with advanced computing technologies.

Suggested Citation

  • Su, Z.C. & Chow, Andy H.F. & Fang, C.L. & Liang, E.M. & Zhong, R.X., 2023. "Hierarchical control for stochastic network traffic with reinforcement learning," Transportation Research Part B: Methodological, Elsevier, vol. 167(C), pages 196-216.
    • Handle: RePEc:eee:transb:v:167:y:2023:i:c:p:196-216
    DOI: 10.1016/j.trb.2022.12.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0191261522002028
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.trb.2022.12.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Haddad, Jack & Zheng, Zhengfei, 2020. "Adaptive perimeter control for multi-region accumulation-based models with state delays," Transportation Research Part B: Methodological, Elsevier, vol. 137(C), pages 133-153.
    2. Geroliminis, Nikolas & Daganzo, Carlos F., 2008. "Existence of urban-scale macroscopic fundamental diagrams: Some experimental findings," Transportation Research Part B: Methodological, Elsevier, vol. 42(9), pages 759-770, November.
    3. Daganzo, Carlos F. & Gayah, Vikash V. & Gonzales, Eric J., 2011. "Macroscopic relations of urban traffic variables: Bifurcations, multivaluedness and instability," Transportation Research Part B: Methodological, Elsevier, vol. 45(1), pages 278-288, January.
    4. Ramezani, Mohsen & Haddad, Jack & Geroliminis, Nikolas, 2015. "Dynamics of heterogeneity in urban networks: aggregated traffic modeling and hierarchical control," Transportation Research Part B: Methodological, Elsevier, vol. 74(C), pages 1-19.
    5. Zhong, R.X. & Huang, Y.P. & Chen, C. & Lam, W.H.K. & Xu, D.B. & Sumalee, A., 2018. "Boundary conditions and behavior of the macroscopic fundamental diagram based network traffic dynamics: A control systems perspective," Transportation Research Part B: Methodological, Elsevier, vol. 111(C), pages 327-355.
    6. Huang, Y.P. & Xiong, J.H. & Sumalee, A. & Zheng, N. & Lam, W.H.K. & He, Z.B. & Zhong, R.X., 2020. "A dynamic user equilibrium model for multi-region macroscopic fundamental diagram systems with time-varying delays," Transportation Research Part B: Methodological, Elsevier, vol. 131(C), pages 1-25.
    7. Ying, Cheng-shuo & Chow, Andy H.F. & Chin, Kwai-Sang, 2020. "An actor-critic deep reinforcement learning approach for metro train scheduling with rolling stock circulation under stochastic demand," Transportation Research Part B: Methodological, Elsevier, vol. 140(C), pages 210-235.
    8. Haddad, Jack & Mirkin, Boris, 2020. "Resilient perimeter control of macroscopic fundamental diagram networks under cyberattacks," Transportation Research Part B: Methodological, Elsevier, vol. 132(C), pages 44-59.
    9. Gao, Shengling & Li, Daqing & Zheng, Nan & Hu, Ruiqi & She, Zhikun, 2022. "Resilient perimeter control for hyper-congested two-region networks with MFD dynamics," Transportation Research Part B: Methodological, Elsevier, vol. 156(C), pages 50-75.
    10. Daganzo, Carlos F., 1994. "The cell transmission model: A dynamic representation of highway traffic consistent with the hydrodynamic theory," Transportation Research Part B: Methodological, Elsevier, vol. 28(4), pages 269-287, August.
    11. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    12. Kouvelas, Anastasios & Saeedmanesh, Mohammadreza & Geroliminis, Nikolas, 2017. "Enhancing model-based feedback perimeter control with data-driven online adaptive optimization," Transportation Research Part B: Methodological, Elsevier, vol. 96(C), pages 26-45.
    13. Chow, Andy H.F. & Lo, Hong K., 2007. "Sensitivity analysis of signal control with physical queuing: Delay derivatives and an application," Transportation Research Part B: Methodological, Elsevier, vol. 41(4), pages 462-477, May.
    14. Gayah, Vikash V. & Gao, Xueyu (Shirley) & Nagle, Andrew S., 2014. "On the impacts of locally adaptive signal control on urban network stability and the Macroscopic Fundamental Diagram," Transportation Research Part B: Methodological, Elsevier, vol. 70(C), pages 255-268.
    15. Daganzo, Carlos F., 2007. "Urban gridlock: Macroscopic modeling and mitigation approaches," Transportation Research Part B: Methodological, Elsevier, vol. 41(1), pages 49-62, January.
    16. Zhang, Lele & Garoni, Timothy M & de Gier, Jan, 2013. "A comparative study of Macroscopic Fundamental Diagrams of arterial road networks governed by adaptive traffic signal systems," Transportation Research Part B: Methodological, Elsevier, vol. 49(C), pages 1-23.
    17. Zhong, R.X. & Chen, C. & Huang, Y.P. & Sumalee, A. & Lam, W.H.K. & Xu, D.B., 2018. "Robust perimeter control for two urban regions with macroscopic fundamental diagrams: A control-Lyapunov function approach," Transportation Research Part B: Methodological, Elsevier, vol. 117(PB), pages 687-707.
    18. Haddad, Jack & Geroliminis, Nikolas, 2012. "On the stability of traffic perimeter control in two-region urban cities," Transportation Research Part B: Methodological, Elsevier, vol. 46(9), pages 1159-1176.
    19. Chow, Andy H.F. & Li, Shuai & Zhong, Renxin, 2017. "Multi-objective optimal control formulations for bus service reliability with traffic signals," Transportation Research Part B: Methodological, Elsevier, vol. 103(C), pages 248-268.
    20. Geroliminis, Nikolas & Sun, Jie, 2011. "Hysteresis phenomena of a Macroscopic Fundamental Diagram in freeway networks," Transportation Research Part A: Policy and Practice, Elsevier, vol. 45(9), pages 966-979, November.
    21. Keyvan-Ekbatani, Mehdi & Kouvelas, Anastasios & Papamichail, Ioannis & Papageorgiou, Markos, 2012. "Exploiting the fundamental diagram of urban networks for feedback-based gating," Transportation Research Part B: Methodological, Elsevier, vol. 46(10), pages 1393-1403.
    22. Li, Ye & Mohajerpoor, Reza & Ramezani, Mohsen, 2021. "Perimeter control with real-time location-varying cordon," Transportation Research Part B: Methodological, Elsevier, vol. 150(C), pages 101-120.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ma, Wenfei & Huang, Yunping & Jin, Xiao & Zhong, Renxin, 2024. "Functional form selection and calibration of macroscopic fundamental diagrams," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 640(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gao, Shengling & Li, Daqing & Zheng, Nan & Hu, Ruiqi & She, Zhikun, 2022. "Resilient perimeter control for hyper-congested two-region networks with MFD dynamics," Transportation Research Part B: Methodological, Elsevier, vol. 156(C), pages 50-75.
    2. Guo, Yajuan & Yang, Licai & Hao, Shenxue & Gu, Xinxin, 2021. "Perimeter traffic control for single urban congested region with macroscopic fundamental diagram and boundary conditions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 562(C).
    3. Ding, Heng & Di, Yunran & Feng, Zhongxiang & Zhang, Weihua & Zheng, Xiaoyan & Yang, Tao, 2022. "A perimeter control method for a congested urban road network with dynamic and variable ranges," Transportation Research Part B: Methodological, Elsevier, vol. 155(C), pages 160-187.
    4. Haddad, Jack & Zheng, Zhengfei, 2020. "Adaptive perimeter control for multi-region accumulation-based models with state delays," Transportation Research Part B: Methodological, Elsevier, vol. 137(C), pages 133-153.
    5. Zhong, R.X. & Chen, C. & Huang, Y.P. & Sumalee, A. & Lam, W.H.K. & Xu, D.B., 2018. "Robust perimeter control for two urban regions with macroscopic fundamental diagrams: A control-Lyapunov function approach," Transportation Research Part B: Methodological, Elsevier, vol. 117(PB), pages 687-707.
    6. Alonso, Borja & Ibeas, Ángel & Musolino, Giuseppe & Rindone, Corrado & Vitetta, Antonino, 2019. "Effects of traffic control regulation on Network Macroscopic Fundamental Diagram: A statistical analysis of real data," Transportation Research Part A: Policy and Practice, Elsevier, vol. 126(C), pages 136-151.
    7. Ma, Wenfei & Huang, Yunping & Jin, Xiao & Zhong, Renxin, 2024. "Functional form selection and calibration of macroscopic fundamental diagrams," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 640(C).
    8. Ampountolas, Konstantinos & Zheng, Nan & Geroliminis, Nikolas, 2017. "Macroscopic modelling and robust control of bi-modal multi-region urban road networks," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 616-637.
    9. Haddad, Jack, 2017. "Optimal perimeter control synthesis for two urban regions with aggregate boundary queue dynamics," Transportation Research Part B: Methodological, Elsevier, vol. 96(C), pages 1-25.
    10. Haddad, Jack & Ramezani, Mohsen & Geroliminis, Nikolas, 2013. "Cooperative traffic control of a mixed network with two urban regions and a freeway," Transportation Research Part B: Methodological, Elsevier, vol. 54(C), pages 17-36.
    11. Zhong, R.X. & Huang, Y.P. & Chen, C. & Lam, W.H.K. & Xu, D.B. & Sumalee, A., 2018. "Boundary conditions and behavior of the macroscopic fundamental diagram based network traffic dynamics: A control systems perspective," Transportation Research Part B: Methodological, Elsevier, vol. 111(C), pages 327-355.
    12. Ramezani, Mohsen & Haddad, Jack & Geroliminis, Nikolas, 2015. "Dynamics of heterogeneity in urban networks: aggregated traffic modeling and hierarchical control," Transportation Research Part B: Methodological, Elsevier, vol. 74(C), pages 1-19.
    13. Yang, Lei & Yin, Suwan & Han, Ke & Haddad, Jack & Hu, Minghua, 2017. "Fundamental diagrams of airport surface traffic: Models and applications," Transportation Research Part B: Methodological, Elsevier, vol. 106(C), pages 29-51.
    14. Kouvelas, Anastasios & Saeedmanesh, Mohammadreza & Geroliminis, Nikolas, 2017. "Enhancing model-based feedback perimeter control with data-driven online adaptive optimization," Transportation Research Part B: Methodological, Elsevier, vol. 96(C), pages 26-45.
    15. Wada, Kentaro & Satsukawa, Koki & Smith, Mike & Akamatsu, Takashi, 2019. "Network throughput under dynamic user equilibrium: Queue spillback, paradox and traffic control," Transportation Research Part B: Methodological, Elsevier, vol. 126(C), pages 391-413.
    16. Zhang, Zhao & Parr, Scott A. & Jiang, Hai & Wolshon, Brian, 2015. "Optimization model for regional evacuation transportation system using macroscopic productivity function," Transportation Research Part B: Methodological, Elsevier, vol. 81(P2), pages 616-630.
    17. Mohajerpoor, Reza & Saberi, Meead & Vu, Hai L. & Garoni, Timothy M. & Ramezani, Mohsen, 2020. "H∞ robust perimeter flow control in urban networks with partial information feedback," Transportation Research Part B: Methodological, Elsevier, vol. 137(C), pages 47-73.
    18. Guo, Qiangqiang & Ban, Xuegang (Jeff), 2020. "Macroscopic fundamental diagram based perimeter control considering dynamic user equilibrium," Transportation Research Part B: Methodological, Elsevier, vol. 136(C), pages 87-109.
    19. Gayah, Vikash V. & Gao, Xueyu (Shirley) & Nagle, Andrew S., 2014. "On the impacts of locally adaptive signal control on urban network stability and the Macroscopic Fundamental Diagram," Transportation Research Part B: Methodological, Elsevier, vol. 70(C), pages 255-268.
    20. Haddad, Jack & Shraiber, Arie, 2014. "Robust perimeter control design for an urban region," Transportation Research Part B: Methodological, Elsevier, vol. 68(C), pages 315-332.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:transb:v:167:y:2023:i:c:p:196-216. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/548/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.