Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage without interference over long times. A key motivation for such research is to perform classic computation tasks, such as parsing. However, memory-augmented neural networks are notoriously hard to train, requiring many backpropagation epochs and a lot of data. In this paper, we introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages and circumvents the training problem by training only the output layer of a recurrent net and employing auxiliary information during training about the desired interaction with a stack. In our experiments, we validate the reservoir stack machine against deep and shallow networks from the literature on three benchmark tasks for Neural Turing machines and six deterministic context-free languages. Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data, requiring only a few seconds of training time and 100 training sequences.
翻译:内存强化神经网络 内存强化神经网络 配置一个具有明确记忆的经常性神经网络, 以支持需要长期不干扰信息存储的任务 。 这种研究的一个主要动机是执行典型的计算任务, 例如 解析 。 然而, 内存强化神经网络 的训练非常困难, 需要许多反向分析器和大量数据 。 在本文中, 我们引入了储油层堆积机, 这个模型可以明显地识别所有确定性的环境无语言, 并绕过培训问题, 其方法是只培训一个经常性网络的输出层, 并在培训过程中使用关于想要与堆叠互动的辅助信息 。 在我们的实验中, 我们验证储油层堆积机, 与文献中关于神经图解机器和六种确定性背景语言的三项基准任务的深浅网络 。 我们的结果表明, 储油层堆积机在测试序列比培训数据长, 只需要几秒钟的培训时间和100个培训序列 。