Factored Online Planning in Many-Agent POMDPs

In centralized multi-agent systems, often modeled as multi-agent partially observable Markov decision processes (MPOMDPs), the action and observation spaces grow exponentially with the number of agents, making the value and belief state estimation of single-agent online planning ineffective. Prior work partially tackles value estimation by exploiting the inherent structure of multi-agent settings via so-called coordination graphs. Additionally, belief state estimation has been improved by incorporating the likelihood of observations into the approximation. However, the challenges of value estimation and state estimation have only been tackled individually, which prevents these methods from scaling to many agents. Therefore, we address these challenges simultaneously. First, we introduce weighted particle filtering to sample-based online planners in MPOMDPs. Second, we present a scalable approximation of the belief state. Third, we bring an approach that exploits the typical locality of agent interactions to novel online planning algorithms for MPOMDPs operating on a so-called sparse particle filter belief tree. Our algorithms show competitive performance for settings with only a few agents and outperform state-of-the-art algorithms on benchmarks with many agents.