Noisy-MAPPO: Noisy Advantage Values for Cooperative Multi-agent Actor-Critic methods

Multi-Agent Reinforcement Learning (MARL) has seen revolutionary breakthroughs with its successful application to multi-agent cooperative tasks such as robot swarms control, autonomous vehicle coordination, and computer games. In this paper, we propose Noisy-MAPPO, which achieves more than 90\% winning rates in all StarCraft Multi-agent Challenge (SMAC) scenarios. First, we theoretically generalize Proximal Policy Optimization (PPO) to Multi-agent PPO (MAPPO) by lower bound of Trust Region Policy Optimization (TRPO). However, we find the shared advantage values in such MAPPO objective function may mislead the learning of some agents, which are not related to these advantage values, called \textit{The Policies Overfitting in Multi-agent Cooperation(POMAC)}. Therefore, we propose noise advatange-value methods (Noisy-MAPPO and Advantage-Noisy-MAPPO) to solve this problem. The experimental results show that our random noise method improves the performance of vanilla MAPPO by 80\% in some Super-Hard scenarios in SMAC. We open-source the code at \url{https://github.com/hijkzzz/noisy-mappo}.