MARLEM: A multi-agent reinforcement learning simulation framework for implicit cooperation in decentralized local energy markets

The proliferation of distributed energy resources requires advanced coordination mechanisms for Local Energy Markets (LEMs) to maintain grid stability and economic efficiency, a challenge ill-suited to traditional centralized control. A significant research gap exists due to the lack of simulation frameworks that cohesively integrate realistic, decentralized market dynamics, physical grid constraints, and advanced Multi-Agent Reinforcement Learning (MARL) capabilities, especially for studying emergent coordination under partial observability. This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents’ observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem), we show the framework’s ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.