Coordination is one of the most difficult aspects of multi-agent reinforcement learning (MARL). One reason is that agents normally choose their actions independently of one another. In order to see coordination strategies emerging from the combination of independent policies, the recent research has focused on the use of a centralized function (CF) that learns each agent's contribution to the team reward. However, the structure in which the environment is presented to the agents and to the CF is typically overlooked. We have observed that the features used to describe the coordination problem can be represented as vertex features of a latent graph structure. Here, we present TransfQMix, a new approach that uses transformers to leverage this latent structure and learn better coordination policies. Our transformer agents perform a graph reasoning over the state of the observable entities. Our transformer Q-mixer learns a monotonic mixing-function from a larger graph that includes the internal and external states of the agents. TransfQMix is designed to be entirely transferable, meaning that same parameters can be used to control and train larger or smaller teams of agents. This enables to deploy promising approaches to save training time and derive general policies in MARL, such as transfer learning, zero-shot transfer, and curriculum learning. We report TransfQMix's performances in the Spread and StarCraft II environments. In both settings, it outperforms state-of-the-art Q-Learning models, and it demonstrates effectiveness in solving problems that other methods can not solve.
翻译:协调是多试剂强化学习(MARL)最困难的方面之一。一个原因是代理商通常各自独立地选择行动。为了看到独立政策的组合所产生的协调战略,最近的研究侧重于使用中央功能(CF),了解每个代理商对团队奖赏的贡献。然而,向代理商和CF展示环境的结构通常被忽视。我们观察到,描述协调问题所使用的特征可以作为潜在图形结构的顶点特征。在这里,我们介绍 TransfQMix,一种使用变异器利用这一潜在结构并学习更好的协调政策的新方法。我们的变异器代理商对可观测实体的状况进行图表推理。我们的变异器Q-Mix从包含代理商内部和外部状态的更大图表中学习一个单调混合功能。 TransfQMix的设计是完全可转让的,这意味着可以使用相同的参数来控制并培训规模或较小的代理商团队。这样可以部署有希望的方法来节省培训时间,并在MAR-C的设置中制定一般政策。我们变异模型,在MAR-Q中,我们学习S-Tradingstal-Trading Stal-modrof lish