Attention-Based Recurrency for Multi-Agent Reinforcement Learning under State Uncertainty

State uncertainty poses a major challenge for decentralized coordination, where multiple agents act according to noisy observations without any access to other agents' information. Centralized training for decentralized execution (CTDE) is a multi-agent reinforcement learning paradigm, which exploits global information to learn a centralized value function in order to derive coordinated multi-agent policies. State-based CTDE has become popular in multi-agent research due to remarkable progress in the StarCraft Multi-Agent Challenge (SMAC). However, SMAC scenarios are less suited for evaluation against state uncertainty due to deterministic observations and low variance in initial states. Furthermore, state-based CTDE largely neglects state uncertainty w.r.t. decentralization of agents and observations thus being possibly less effective in more general settings. In this paper, we address state uncertainty and introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states to provide a more general and configurable benchmark. We then propose Attention-based Embeddings of Recurrency In multi-Agent Learning (AERIAL) to approximate value functions under consideration of state uncertainty. AERIAL replaces the true state in CTDE with the memory representation of all agents' recurrent functions, which are processed by a self-attention mechanism. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. We also evaluate the robustness of AERIAL and state-based CTDE against various configurations of state uncertainty in MessySMAC.