Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.
翻译:变异模型是变异模型。 要提供输入符、 位置和分块嵌入器的顺序和类型信息, 通常会添加到输入中 。 最近的工作提议了位置编码的变异, 相对位置编码的变异, 取得更好的性能。 我们的分析表明, 增益实际上是从输入中移动位置信息到注意层。 受此驱动, 我们引入了变异器脱钩定位注意( DIET), 这是将位置和分段信息编码到变异器模型中的简单而有效的机制 。 拟议的方法有更快的培训和推断时间, 同时在 GLUE、 XTREME 和 WMT 基准上实现竞争性性能。 我们进一步推广了远程变异器和显示性能增益的方法 。