Communication enables agents to cooperate to achieve their goals. Learning when to communicate, i.e. sparse communication, is particularly important where bandwidth is limited, in situations where agents interact with humans, in partially observable scenarios where agents must convey information unavailable to others, and in non-cooperative scenarios where agents may hide information to gain a competitive advantage. Recent work in learning sparse communication, however, suffers from high variance training where, the price of decreasing communication is a decrease in reward, particularly in cooperative tasks. Sparse communications are necessary to match agent communication to limited human bandwidth. Humans additionally communicate via discrete linguistic tokens, previously shown to decrease task performance when compared to continuous communication vectors. This research addresses the above issues by limiting the loss in reward of decreasing communication and eliminating the penalty for discretization. In this work, we successfully constrain training using a learned gate to regulate when to communicate while using discrete prototypes that reflect what to communicate for cooperative tasks with partial observability. We provide two types of "Enforcers" for hard and soft budget constraints and present results of communication under different budgets. We show that our method satisfies constraints while yielding the same performance as comparable, unconstrained methods.
翻译:在带宽有限的情况下,在代理人与人类互动的情况下,在部分可见的情况下,在代理人必须传递他人无法获得的信息的情况下,在代理人必须传递他人无法获取的信息的非合作情况下,以及在代理人可能隐瞒信息以获得竞争优势的非合作情况下,学习通信的近期工作使通信缺乏,但由于培训差异很大,通信减少的代价是奖励的降低,特别是在合作任务方面,通信减少的代价是报酬的降低。为了将代理通信与有限的人类带宽相匹配,通信不全面是必要的。在与连续通信载体相比,以往显示任务性能下降的情况下,人类通过离散语言象征进行的额外通信。这一研究解决了上述问题,通过减少对通信的奖励损失和消除对离散化的处罚来限制上述问题。在这项工作中,我们成功地限制了培训,在使用离散的原型来调节通信时,同时使用反映部分可遵守的合作任务的信息。我们为硬软预算限制和在不同预算下显示通信结果提供了两种“强制”的“强制”。我们的方法满足了限制,同时产生类似的、不严谨的方法。