Nowadays artificial neural network models achieve remarkable results in many disciplines. Functions mapping the representation provided by the model to the probability distribution are the inseparable aspect of deep learning solutions. Although softmax is a commonly accepted probability mapping function in the machine learning community, it cannot return sparse outputs and always spreads the positive probability to all positions. In this paper, we propose r-softmax, a modification of the softmax, outputting sparse probability distribution with controllable sparsity rate. In contrast to the existing sparse probability mapping functions, we provide an intuitive mechanism for controlling the output sparsity level. We show on several multi-label datasets that r-softmax outperforms other sparse alternatives to softmax and is highly competitive with the original softmax. We also apply r-softmax to the self-attention module of a pre-trained transformer language model and demonstrate that it leads to improved performance when fine-tuning the model on different natural language processing tasks.
翻译:现在,人工神经网络模型在许多学科中取得了显著的成果。将模型提供的表示映射为概率分布的函数是深度学习解决方案的不可分割的方面。虽然 softmax 是机器学习社区中通常接受的概率映射函数,但它不能返回稀疏的输出,并且始终将正概率分散到所有位置。在本文中,我们提出了 r-softmax,一种修改的 softmax,它输出可控稀疏率的稀疏概率分布。与现有的稀疏概率映射函数不同,我们提供了一种直观的机制来控制输出的稀疏程度。我们在几个多标签数据集上展示了 r-softmax 超越了 softmax 的其他稀疏替代方案,并与原始的 softmax 竞争力强。我们还将 r-softmax 应用于经过预训练的 transformer 语言模型的自我注意模块,证明它在不同的自然语言处理任务的微调中可以导致性能改善。