It is well known that recurrent neural networks (RNNs) faced limitations in learning long-term dependencies that have been addressed by memory structures in long short-term memory (LSTM) networks. Matrix neural networks feature matrix representation which inherently preserves the spatial structure of data and has the potential to provide better memory structures when compared to canonical neural networks that use vector representation. Neural Turing machines (NTMs) are novel RNNs that implement notion of programmable computers with neural network controllers to feature algorithms that have copying, sorting, and associative recall tasks. In this paper, we study the augmentation of memory capacity with a matrix representation of RNNs and NTMs (MatNTMs). We investigate if matrix representation has a better memory capacity than the vector representations in conventional neural networks. We use a probabilistic model of the memory capacity using Fisher information and investigate how the memory capacity for matrix representation networks are limited under various constraints, and in general, without any constraints. In the case of memory capacity without any constraints, we found that the upper bound on memory capacity to be $N^2$ for an $N\times N$ state matrix. The results from our experiments using synthetic algorithmic tasks show that MatNTMs have a better learning capacity when compared to its counterparts.
翻译:众所周知,经常性神经网络(RNN)在学习长期依赖性方面受到限制,长期记忆结构在短期记忆(LSTM)网络中的记忆结构已经解决了这一问题。矩阵神经网络以矩阵代表形式显示记忆能力的增强,这在本质上保护了数据的空间结构,并有可能提供更好的记忆结构,而与使用病媒代表的罐体神经网络相比,它具有更好的记忆结构。神经涡轮机(NTMs)是新颖的RNNN,它实施具有神经网络控制器的可编程计算机概念,以描述具有复制、排序和连带召回任务的算法。在本文中,我们用RNNTM(M)和NTM(M)的矩阵代表形式来研究记忆能力的增强。我们调查矩阵代表是否比传统神经网络中的矢量代表具有更好的记忆能力。我们使用渔业信息进行记忆能力的概率模型,并调查矩阵代表网络的记忆能力在各种制约下如何不受约束,一般而言,没有任何限制。在记忆能力方面,我们发现在没有任何限制的情况下,我们的记忆能力上限是:我们从存储能力上限上从$N2美元到用我们的记忆能力在合成矩阵上学习一个比较的矩阵时,而能显示其矩阵的能力是用来进行合成矩阵。