Offline multi-agent reinforcement learning (MARL) is severely hampered by the challenge of evaluating out-of-distribution (OOD) joint actions. Our core finding is that when the behavior policy is factorized - a common scenario where agents act fully or partially independently during data collection - a strategy of partial action replacement (PAR) can significantly mitigate this challenge. PAR updates a single or part of agents' actions while the others remain fixed to the behavioral data, reducing distribution shift compared to full joint-action updates. Based on this insight, we develop Soft-Partial Conservative Q-Learning (SPaCQL), using PAR to mitigate OOD issue and dynamically weighting different PAR strategies based on the uncertainty of value estimation. We provide a rigorous theoretical foundation for this approach, proving that under factorized behavior policies, the induced distribution shift scales linearly with the number of deviating agents rather than exponentially with the joint-action space. This yields a provably tighter value error bound for this important class of offline MARL problems. Our theoretical results also indicate that SPaCQL adaptively addresses distribution shift using uncertainty-informed weights. Our empirical results demonstrate SPaCQL enables more effective policy learning, and manifest its remarkable superiority over baseline algorithms when the offline dataset exhibits the independence structure.
翻译：离线多智能体强化学习（MARL）因评估分布外（OOD）联合动作的挑战而受到严重制约。我们的核心发现是，当行为策略具有因子化特性——即数据收集过程中智能体完全或部分独立行动的常见场景时，采用部分动作替换（PAR）策略可显著缓解这一挑战。PAR 在更新单个或部分智能体动作的同时，保持其他智能体动作与行为数据一致，相比完全联合动作更新，有效降低了分布偏移。基于这一洞见，我们提出了软部分保守 Q 学习（SPaCQL），利用 PAR 缓解 OOD 问题，并根据价值估计的不确定性动态加权不同的 PAR 策略。我们为此方法提供了严格的理论基础，证明了在因子化行为策略下，所引发的分布偏移随偏离智能体数量线性增长，而非随联合动作空间指数增长，从而为这类重要的离线 MARL 问题提供了可证明的更紧的价值误差界。我们的理论结果还表明，SPaCQL 能利用不确定性感知权重自适应地处理分布偏移。实验结果表明，SPaCQL 实现了更有效的策略学习，并在离线数据集呈现独立性结构时，显著优于基线算法。
相关内容
微信扫码咨询专知VIP会员