Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.
翻译：鉴于大门的经常性单元的成功,一个自然的问题是,长期短期内存(LSTM)网络的所有大门是否都有必要。先前的研究显示,忘记大门是LSTM中最重要的大门之一。在这里,我们显示,只有忘记大门的LSTM版本,带有时序初始偏差,不仅提供了计算节余,而且超过了多个基准数据集的标准LSTM,并与一些当代最佳模型竞争。我们提议的网络JANET在MNIST和PMNIST数据集上实现了99%和92.5%的默认,超过了标准LSTM,该标准生成98.5%和91%的默认值。
相关内容
微信扫码咨询专知VIP会员