Buildings-to-grid with generalized energy storage: A multi-agent decomposed deep reinforcement learning approach for delayed rewards

The growing penetration of distributed renewable energy and flexible building loads is intensifying the bidirectional building-to-grid (BtG) coupling. However, the inherent heterogeneity between electrochemical batteries and comfort-coupled thermal storage complicates coordinated control. To bridge this gap, the present study proposes a generalized energy storage system (GESS) that represents both devices with a common state of charge and generalized charge/discharge power. An adaptive self-loss term captures both battery self-discharge and temperature-dependent passive heat exchange. The model maps the generalized power of thermal energy storage to equivalent electrical power, while accounting for the thermal inertia of internal spaces and heating, ventilation, and air-conditioning systems. To address delayed rewards in the GESS, a multi-agent decomposed deep reinforcement learning approach is developed. The control problem is formulated as a sequential partially observable Markov decision process with a dual-critic architecture that redistributes immediate rewards to construct delayed rewards. Decentralized actors are optimized using a clipped surrogate objective with combined advantage estimates and control variate stabilization. Numerical experiments on the test system demonstrate that the proposed method enhances building profitability and reduces grid operating costs.