Humans communicate with graphical sketches apart from symbolic languages. Primarily focusing on the latter, recent studies of emergent communication overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity. In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions. To inspect the emerged conventions, we define three fundamental properties -- iconicity, symbolicity, and semanticity -- and design evaluation methods accordingly. Our experimental results under different controls are consistent with the observation in studies of human graphical conventions. Of note, we find that evolved sketches can preserve the continuum of semantics under proper environmental pressures. More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents. We hope the present research can pave the path for studying emergent communication with the modality of sketches.
翻译:人类与图形素描除象征性语言外进行交流。 以后者为主, 最近对新兴通信的研究忽略了这些素描; 它们没有说明在符号性和象征性之间的权衡中出现符号标志系统的进化过程。 在这项工作中,我们迈出了第一步,通过两个神经剂来模拟和模拟这个过程, 玩一个视觉通信游戏; 发送者通过在画布上绘制图谱与接收者进行交流。 我们设计了一种新型强化学习方法, 使代理商能够共同发展成成功的通信和抽象的图形公约。 为了检查新产生的公约, 我们定义了三种基本特性 -- -- 标志性、象征性和语义性 -- 并据此设计了评价方法。 我们在不同控制下实验的结果与人类图形学研究中观察的结果是一致的。 值得注意的是, 我们发现进化的素描可以保存在适当的环境压力下的语义学连续体。 更有趣的是, 共同变化的代理商可以根据他们对参考者的熟悉程度, 而在常规和标志性通信之间转换。 我们希望目前的研究能够铺设研究以素描写方式研究新通信的路径。</s>