IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v14y2022i21p14590-d964839.html
   My bibliography  Save this article

Adaptive Deep Q-Network Algorithm with Exponential Reward Mechanism for Traffic Control in Urban Intersection Networks

Author

Listed:
  • Muhammad Riza Tanwirul Fuad

    (Department of Engineering Physics, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia)

  • Eric Okto Fernandez

    (Department of Engineering Physics, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia)

  • Faqihza Mukhlish

    (Engineering Physics Research Group, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia)

  • Adiyana Putri

    (Graduate Program of Engineering Physics, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia)

  • Herman Yoseph Sutarto

    (Department of Intelligent System, PT. Pusat Riset Energi, Bandung 40226, Indonesia)

  • Yosi Agustina Hidayat

    (Industrial System and Techno-Economy Research Group, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia)

  • Endra Joelianto

    (Instrumentation and Control Research Group, Faculty of Industrial Technology, Institut Teknologi Bandung, Bandung 40132, Indonesia
    University Center of Excellence Artificial Intelligence on Vision, NLP and Big Data Analytics (U-CoE AI-VLB), Institut Teknologi Bandung, Bandung 40132, Indonesia)

Abstract

The demand for transportation has increased significantly in recent decades in line with the increasing demand for passenger and freight mobility, especially in urban areas. One of the most negative impacts is the increasing level of traffic congestion. A possible short-term solution to solve this problem is to utilize a traffic control system. However, most traffic control systems still use classical control algorithms with the green phase sequence determined, based on a specific strategy. Studies have proven that this approach does not provide the expected congestion solution. In this paper, an adaptive traffic controller was developed that uses a reinforcement learning algorithm called deep Q-network (DQN). Since the DQN performance is determined by reward selection, an exponential reward function, based on the macroscopic fundamental diagram (MFD) of the distribution of vehicle density at intersections was considered. The action taken by the DQN is determining traffic phases, based on various rewards, ranging from pressure to adaptive loading of pressure and queue length. The reinforcement learning algorithm was then applied to the SUMO traffic simulation software to assess the effectiveness of the proposed strategy. The DQN-based control algorithm with the adaptive reward mechanism achieved the best performance with a vehicle throughput of 56,384 vehicles, followed by the classical and conventional control methods, such as Webster (50,366 vehicles), max-pressure (50,541 vehicles) and uniform (46,241 vehicles) traffic control. The significant increase in vehicle throughput achieved by the adaptive DQN-based control algorithm with an exponential reward mechanism means that the proposed traffic control could increase the area productivity, implying that the intersections could accommodate more vehicles so that the possibility of congestion was reduced. The algorithm performed remarkably in preventing congestion in a traffic network model of Central Jakarta as one of the world’s most congested cities. This result indicates that traffic control design using MFD as a performance measure can be a successful future direction in the development of reinforcement learning for traffic control systems.

Suggested Citation

  • Muhammad Riza Tanwirul Fuad & Eric Okto Fernandez & Faqihza Mukhlish & Adiyana Putri & Herman Yoseph Sutarto & Yosi Agustina Hidayat & Endra Joelianto, 2022. "Adaptive Deep Q-Network Algorithm with Exponential Reward Mechanism for Traffic Control in Urban Intersection Networks," Sustainability, MDPI, vol. 14(21), pages 1-20, November.
  • Handle: RePEc:gam:jsusta:v:14:y:2022:i:21:p:14590-:d:964839
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/14/21/14590/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2071-1050/14/21/14590/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gayah, Vikash V. & Gao, Xueyu (Shirley) & Nagle, Andrew S., 2014. "On the impacts of locally adaptive signal control on urban network stability and the Macroscopic Fundamental Diagram," Transportation Research Part B: Methodological, Elsevier, vol. 70(C), pages 255-268.
    2. Geroliminis, Nikolas & Daganzo, Carlos F., 2008. "Existence of urban-scale macroscopic fundamental diagrams: Some experimental findings," Transportation Research Part B: Methodological, Elsevier, vol. 42(9), pages 759-770, November.
    3. S. A. Ramadhan & H. Y. Sutarto & G. S. Kuswana & E. Joelianto, 2020. "Application of area traffic control using the max-pressure algorithm," Transportation Planning and Technology, Taylor & Francis Journals, vol. 43(8), pages 783-802, November.
    4. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    5. Fei Yan & Fu-li Tian & Zhong-ke Shi, 2015. "Iterative Learning Control Approach for Signaling Split in Urban Traffic Networks with Macroscopic Fundamental Diagrams," Mathematical Problems in Engineering, Hindawi, vol. 2015, pages 1-12, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Su, Z.C. & Chow, Andy H.F. & Fang, C.L. & Liang, E.M. & Zhong, R.X., 2023. "Hierarchical control for stochastic network traffic with reinforcement learning," Transportation Research Part B: Methodological, Elsevier, vol. 167(C), pages 196-216.
    2. Kouvelas, Anastasios & Saeedmanesh, Mohammadreza & Geroliminis, Nikolas, 2017. "Enhancing model-based feedback perimeter control with data-driven online adaptive optimization," Transportation Research Part B: Methodological, Elsevier, vol. 96(C), pages 26-45.
    3. Ampountolas, Konstantinos & Zheng, Nan & Geroliminis, Nikolas, 2017. "Macroscopic modelling and robust control of bi-modal multi-region urban road networks," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 616-637.
    4. Niu, Xiao-Jing & Zhao, Xiao-Mei & Xie, Dong-Fan & Liu, Feng & Bi, Jun & Lu, Chaoru, 2022. "Impact of large-scale activities on macroscopic fundamental diagram: Field data analysis and modeling," Transportation Research Part A: Policy and Practice, Elsevier, vol. 161(C), pages 241-268.
    5. Guo, Qiangqiang & Ban, Xuegang (Jeff), 2020. "Macroscopic fundamental diagram based perimeter control considering dynamic user equilibrium," Transportation Research Part B: Methodological, Elsevier, vol. 136(C), pages 87-109.
    6. Alonso, Borja & Ibeas, Ángel & Musolino, Giuseppe & Rindone, Corrado & Vitetta, Antonino, 2019. "Effects of traffic control regulation on Network Macroscopic Fundamental Diagram: A statistical analysis of real data," Transportation Research Part A: Policy and Practice, Elsevier, vol. 126(C), pages 136-151.
    7. Laval, Jorge A. & Castrillón, Felipe, 2015. "Stochastic approximations for the macroscopic fundamental diagram of urban networks," Transportation Research Part B: Methodological, Elsevier, vol. 81(P3), pages 904-916.
    8. Gao, Shengling & Li, Daqing & Zheng, Nan & Hu, Ruiqi & She, Zhikun, 2022. "Resilient perimeter control for hyper-congested two-region networks with MFD dynamics," Transportation Research Part B: Methodological, Elsevier, vol. 156(C), pages 50-75.
    9. Liu, Wei & Szeto, Wai Yuen, 2020. "Learning and managing stochastic network traffic dynamics with an aggregate traffic representation," Transportation Research Part B: Methodological, Elsevier, vol. 137(C), pages 19-46.
    10. Amirgholy, Mahyar & Shahabi, Mehrdad & Gao, H. Oliver, 2017. "Optimal design of sustainable transit systems in congested urban networks: A macroscopic approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 103(C), pages 261-285.
    11. Ramezani, Mohsen & Haddad, Jack & Geroliminis, Nikolas, 2015. "Dynamics of heterogeneity in urban networks: aggregated traffic modeling and hierarchical control," Transportation Research Part B: Methodological, Elsevier, vol. 74(C), pages 1-19.
    12. Wang, Yi & Szeto, W.Y. & Han, Ke & Friesz, Terry L., 2018. "Dynamic traffic assignment: A review of the methodological advances for environmentally sustainable road transportation applications," Transportation Research Part B: Methodological, Elsevier, vol. 111(C), pages 370-394.
    13. Wu, Chao-Yun & Li, Ming & Jiang, Rui & Hao, Qing-Yi & Hu, Mao-Bin, 2018. "Perimeter control for urban traffic system based on macroscopic fundamental diagram," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 503(C), pages 231-242.
    14. Knoop, Victor L. & van Lint, Hans & Hoogendoorn, Serge P., 2015. "Traffic dynamics: Its impact on the Macroscopic Fundamental Diagram," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 438(C), pages 236-250.
    15. Xu, Guanhao & Gayah, Vikash V., 2023. "Non-unimodal and non-concave relationships in the network Macroscopic Fundamental Diagram caused by hierarchical streets," Transportation Research Part B: Methodological, Elsevier, vol. 173(C), pages 203-227.
    16. Gupta, Namrata & Patil, Gopal R. & Vu, Hai L., 2023. "Simple abstract models to study stability of urban networks with decentralized signal control," Transportation Research Part B: Methodological, Elsevier, vol. 172(C), pages 93-116.
    17. Zhong, R.X. & Chen, C. & Huang, Y.P. & Sumalee, A. & Lam, W.H.K. & Xu, D.B., 2018. "Robust perimeter control for two urban regions with macroscopic fundamental diagrams: A control-Lyapunov function approach," Transportation Research Part B: Methodological, Elsevier, vol. 117(PB), pages 687-707.
    18. Zhang, Zhao & Parr, Scott A. & Jiang, Hai & Wolshon, Brian, 2015. "Optimization model for regional evacuation transportation system using macroscopic productivity function," Transportation Research Part B: Methodological, Elsevier, vol. 81(P2), pages 616-630.
    19. Amirgholy, Mahyar & Gao, H. Oliver, 2017. "Modeling the dynamics of congestion in large urban networks using the macroscopic fundamental diagram: User equilibrium, system optimum, and pricing strategies," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 215-237.
    20. Guo, Yajuan & Yang, Licai & Hao, Shenxue & Gu, Xinxin, 2021. "Perimeter traffic control for single urban congested region with macroscopic fundamental diagram and boundary conditions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 562(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:14:y:2022:i:21:p:14590-:d:964839. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.