Cooperative game is a critical research area in the multi-agent reinforcement learning (MARL). Global reward game is a subclass of cooperative games, where all agents aim to maximize the global reward. Credit assignment is an important problem studied in the global reward game. Most of previous works stood by the view of non-cooperative-game theoretical framework with the shared reward approach, i.e., each agent being assigned a shared global reward directly. This, however, may give each agent an inaccurate reward on its contribution to the group, which could cause inefficient learning. To deal with this problem, we i) introduce a cooperative-game theoretical framework called extended convex game (ECG) that is a superset of global reward game, and ii) propose a local reward approach called Shapley Q-value. Shapley Q-value is able to distribute the global reward, reflecting each agent's own contribution in contrast to the shared reward approach. Moreover, we derive an MARL algorithm called Shapley Q-value deep deterministic policy gradient (SQDDPG), using Shapley Q-value as the critic for each agent. We evaluate SQDDPG on Cooperative Navigation, Prey-and-Predator and Traffic Junction, compared with the state-of-the-art algorithms, e.g., MADDPG, COMA, Independent DDPG and Independent A2C. In the experiments, SQDDPG shows a significant improvement on the convergence rate. Finally, we plot Shapley Q-value and validate the property of fair credit assignment.
翻译:合作游戏是多试剂强化学习(MARL)中的一个关键研究领域。全球奖赏游戏是一个合作游戏的子类,所有代理商都致力于最大限度地提高全球奖赏。信用分配是全球奖赏游戏中研究的一个重要问题。以前的大部分作品都以非合作游戏理论框架的观点为基础,采用共享奖赏方法,即每个代理商直接获得共享全球奖赏。然而,这可能给每个代理商带来不准确的奖赏,使其对集团的贡献可能导致低效学习。为了解决这一问题,我们i)引入了一个合作游戏理论框架,称为扩展的康威游戏(ECG),这是全球奖赏游戏的超级组合,并且二)提出一个名为Shapley Q值的本地奖赏方法。Spley Qvalit能够分配全球奖赏,反映每个代理商自己对共享奖赏方法的贡献。此外,我们用沙佩利 QDPL(SQGDGG) 的精度、SDDDL 和SDGADL 的透明性、SDGLA、S-DLADL 和SDLAVAL 和SUDLADLA、S-DLA、S-DLADLA、S-DLADLA、SUD、S-S-DLA、S-DLADLADLADLA、S-S-S-S-GLA、S-S-DLA、S-DLA、S-GLA、S-S-DDLA、S-GLADLADLA、S-DLA、S、S、S、SDLADLA、S、S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-DGL-DLA、S-S-S-S-S-S-DAR-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-DL-S-S-S-S-D