Self-attention (SA), which encodes vector sequences according to their pairwise similarity, is widely used in speech recognition due to its strong context modeling ability. However, when applied to long sequence data, its accuracy is reduced. This is caused by the fact that its weighted average operator may lead to the dispersion of the attention distribution, which results in the relationship between adjacent signals ignored. To address this issue, in this paper, we introduce relative-position-awareness self-attention (RPSA). It not only maintains the global-range dependency modeling ability of self-attention, but also improves the localness modeling ability. Because the local window length of the original RPSA is fixed and sensitive to different test data, here we propose Gaussian-based self-attention (GSA) whose window length is learnable and adaptive to the test data automatically. We further generalize GSA to a new residual Gaussian self-attention (resGSA) for the performance improvement. We apply RPSA, GSA, and resGSA to Transformer-based speech recognition respectively. Experimental results on the AISHELL-1 Mandarin speech recognition corpus demonstrate the effectiveness of the proposed methods. For example, the resGSA-Transformer achieves a character error rate (CER) of 5.86% on the test set, which is relative 7.8% lower than that of the SA-Transformer. Although the performance of the proposed resGSA-Transformer is only slightly better than that of the RPSA-Transformer, it does not have to tune the window length manually.
翻译:自我关注(SA)根据对称相似性对矢量序列进行编码,由于具有很强的环境建模能力,在语音识别中被广泛使用。然而,当应用到长序列数据时,其准确性会降低。这是因为其加权平均操作员可能导致注意力分布分散,从而导致相邻信号之间的关系被忽视。为了解决这一问题,我们在本文件中引入了相对定位意识自留(RSA),它不仅保持了全球范围自控模型的自控能力,而且提高了本地化建模能力。由于原 RPSA 的本地窗口长度是固定的,对不同的测试数据敏感,我们在此提议基于Gausian的加权平均操作员可能会导致注意力分布分散,从而导致相邻信号被忽略。为了改进性能,我们进一步将GSA推广到新的残余自留自留自控(RSAA) 。我们将RPSAA、GSA和ResGSA 的透明性能识别能力稍低一些。