The marriage between mean-field theory and reinforcement learning has shown a great capacity to solve large-scale control problems with homogeneous agents. To break the homogeneity restriction of mean-field theory, a recent interest is to introduce graphon theory to the mean-field paradigm. In this paper, we propose a graphon mean-field control (GMFC) framework to approximate cooperative multi-agent reinforcement learning (MARL) with nonuniform interactions and show that the approximate order is of $\mathcal{O}(\frac{1}{\sqrt{N}})$, with $N$ the number of agents. By discretizing the graphon index of GMFC, we further introduce a smaller class of GMFC called block GMFC, which is shown to well approximate cooperative MARL. Our empirical studies on several examples demonstrate that our GMFC approach is comparable with the state-of-art MARL algorithms while enjoying better scalability.
翻译:平均场理论和强化学习之间的结合显示了解决与同质物剂大规模控制问题的巨大能力。 为打破对平均场理论的同质性限制,最近的一个兴趣是引入平均场范式的图形化理论。 在本文中,我们提议了一个图形化平均场控制框架(GMFC)来将合作性多试剂强化学习(MARL)与非统一的相互作用相近,并表明大约的顺序是$\mathcal{O}(frac{1unsqrt{N ⁇ )$($N$),与物剂的数量相匹配。通过将GMFC的图形化指数分解,我们进一步引入了称为块GMFC的较小的GFC类别,这显示它大致是合作性的ML。我们对几个例子的实验研究表明,我们的GMFC方法与最先进的MARL算法具有可比性,同时具有更好的可缩放性。