Recent studies have demonstrated that the performance of transformers on the task of language modeling obeys a power-law relationship with model size over six orders of magnitude. While transformers exhibit impressive scaling, their performance hinges on processing large amounts of data, and their computational and memory requirements grow quadratically with sequence length. Motivated by these considerations, we construct a Legendre Memory Unit based model that introduces a general prior for sequence processing and exhibits an $O(n)$ and $O(n \ln n)$ (or better) dependency for memory and computation respectively. Over three orders of magnitude, we show that our new architecture attains the same accuracy as transformers with 10x fewer tokens. We also show that for the same amount of training our model improves the loss over transformers about as much as transformers improve over LSTMs. Additionally, we demonstrate that adding global self-attention complements our architecture and the augmented model improves performance even further.
翻译:最近的研究显示,变压器在语言建模任务上的性能符合与6级以上模型大小的电法关系。虽然变压器表现出令人印象深刻的缩放,但其性能取决于大量数据的处理,其计算和内存要求随着序列长度而增长。受这些考虑的驱使,我们建立了一个基于传说记忆股的模型,该模型在顺序处理和显示一般前先引入一个一般的O(n)美元和O(n)n(n)n(n)美元(或更高)美元对记忆和计算的依赖。在3级以上,我们显示我们的新结构获得了与变压器相同的精度,用10x倍的符号。我们还表明,同样程度的模型也提高了变压器的损率,因为变压器比LSTMS改进了多少。 此外,我们证明,增加全球自留装置可以补充我们的结构,而扩大的模型的性能则进一步提高。