Hyperparameter optimization impact and tuning guidelines for decentralized multi-agent reinforcement learning in multi-energy neighborhoods

Driven by the growing integration of distributed energy resources and the rising complexity of energy management, decentralized reinforcement learning (RL) is emerging as a prevalent approach in power and energy systems, offering a model-free and data-efficient solution to address these challenges. One of the challenges posed in implementing RL in energy applications is hyperparameter tuning or optimization, yet systematic studies of it remain scarce. This study systematically explores the role of hyperparameters in determining the computational cost, performance gains, and most influential hyperparameter settings when using multi-agent RL to control electric and thermal energy storage systems in a multi-energy neighborhood. A scalable, decentralized multi-agent reinforcement learning framework based on Proximal Policy Optimization is developed in a custom OpenAI Gym environment to coordinate electric and thermal energy storage within a case study neighborhood with a local grid of six buildings with photovoltaic generation, a wood-chip combined heat and power plant, and both battery and hot-water storage. Scenarios with two to six agents are tested, where hyperparameter optimization yields a 4-5 % variation in performance in grid-purchase reduction, while computational cost per 10 training iterations increases linearly with agent count, underscoring scalability challenges. Global sensitivity analysis identifies learning rate and discount factor as the dominant hyperparameters among these investigations. Partial dependence analysis suggests that learning rates in the range of 1×10⁻³–2×10⁻³ and discount factors between 0.95 and 0.99 yield strong performance for the considered problem formulation. These findings provide actionable guidance for configuring RL-based controllers in neighborhoods with multi-energy systems.