Neural autoregressive sequence models smear the probability among many possible sequences including degenerate ones, such as empty or repetitive sequences. In this work, we tackle one specific case where the model assigns a high probability to unreasonably short sequences. We define the oversmoothing rate to quantify this issue. After confirming the high degree of oversmoothing in neural machine translation, we propose to explicitly minimize the oversmoothing rate during training. We conduct a set of experiments to study the effect of the proposed regularization on both model distribution and decoding performance. We use a neural machine translation task as the testbed and consider three different datasets of varying size. Our experiments reveal three major findings. First, we can control the oversmoothing rate of the model by tuning the strength of the regularization. Second, by enhancing the oversmoothing loss contribution, the probability and the rank of <eos> token decrease heavily at positions where it is not supposed to be. Third, the proposed regularization impacts the outcome of beam search especially when a large beam is used. The degradation of translation quality (measured in BLEU) with a large beam significantly lessens with lower oversmoothing rate, but the degradation compared to smaller beam sizes remains to exist. From these observations, we conclude that the high degree of oversmoothing is the main reason behind the degenerate case of overly probable short sequences in a neural autoregressive model.
翻译:神经自动递减序列模型在包括衰变序列(如空序或重复序列)在内的许多可能序列中折射概率。在这项工作中,我们处理一个具体案例,模型给不合理的短序列分配了高概率。我们定义了超移动率,以量化这一问题。在确认神经机翻译中高度超移动率后,我们提议明确将培训期间超移动率降至最低;我们进行一系列实验,以研究拟议正规化对模型分布和解码性能的影响。我们使用神经机器翻译任务作为测试台,考虑三个不同大小不同的数据集。我们实验揭示了三个主要结果。首先,我们可以通过调整正常化强度来控制模型的超移动率。第二,通过增强超移动损失贡献率的高度,<eos>的概率和级别将大大降低在模型不应存在的位置上。第三,拟议正规化将影响比例搜索的结果,特别是当使用大波束时。翻译质量的退化(在BLEEU中为压低的缩缩缩缩缩缩),与大幅递增的递增比例相比,这些递增的递增的递增比例。