We consider preference communication in two-player multi-objective normal-form games. In such games, the payoffs resulting from joint actions are vector-valued. Taking a utility-based approach, we assume there exists a utility function for each player which maps vectors to scalar utilities and consider agents that aim to maximise the utility of expected payoff vectors. As agents typically do not know their opponent's utility function or strategy, they must learn policies to interact with each other. Inspired by Stackelberg games, we introduce four novel preference communication protocols to aid agents in arriving at adequate solutions. Each protocol describes a specific approach for one agent to communicate preferences over their actions and how another agent responds. Additionally, to study when communication emerges, we introduce a communication protocol where agents must learn when to communicate. These protocols are subsequently evaluated on a set of five benchmark games against baseline agents that do not communicate. We find that preference communication can alter the learning process and lead to the emergence of cyclic policies which had not been previously observed in this setting. We further observe that the resulting policies can heavily depend on the characteristics of the game that is played. Lastly, we find that communication naturally emerges in both cooperative and self-interested settings.
翻译:我们考虑在双玩者多客观的正常形式游戏中进行优先交流。 在这种游戏中,联合行动的回报是矢量价值的。 我们假设每个玩家都有一种实用功能,用来绘制矢量图,以标出目标功率,并考虑旨在尽量扩大预期报酬矢量的效用的代理商。 由于代理商通常不知道对方的效用功能或策略,他们必须学习互动的政策。 在斯塔克尔贝格游戏的启发下,我们引入了四个新的优先交流协议,以帮助代理商达成适当的解决办法。每个协议都描述了一个代理商交流偏好于其行动的具体方法以及另一个代理商的反应方式。此外,在通信出现时,我们引入了一个通信协议,代理商必须学习如何沟通。这些协议随后在一套五种基准游戏上被评估,而不是没有沟通的基线代理商。我们发现,偏好交流可以改变学习过程,导致在这一环境下没有观察到的周期政策的出现。我们进一步指出,由此产生的政策可能在很大程度上取决于游戏的特性。最后,我们发现,通信自然会产生自我和合作关系。