A population-based approach for multi-agent interpretable reinforcement learning

Abstract

Multi-Agent Reinforcement Learning (MARL) made significant progress in the last decade, mainly thanks to the major developments in the field of Deep Neural Networks (DNNs). However, DNNs suffer from a fundamental issue: their lack of interpretability. While this is true for most applications of DNNs, this is exacerbated in their applications in MARL. In fact, the mutual interactions between agents and environment, as well as across agents, make it particularly difficult to understand learned strategies in these settings. One possible way to achieve explainability in MARL is through the use of interpretable models, such as decision trees, that allow for a direct inspection and understanding of their inner workings. In this work, we make a step forward in this direction, proposing a population-based algorithm that combines evolutionary principles with RL for training interpretable models in multi-agent systems. We evaluate the proposed approach in a highly dynamic task where two teams of agents compete with each other. We test different variants of the proposed method in different settings, namely with/without coevolution and with/without initialization from a handcrafted policy. We find that, in most settings, our method is able to find fairly effective policies. Moreover, we show that the learned policies are easy to inspect and, possibly, interpreted based on domain knowledge.