Recurrent neural networks are widely used for modeling spatio-temporal sequences in both nature language processing and neural population dynamics. However, understanding the temporal credit assignment is hard. Here, we propose that each individual connection in the recurrent computation is modeled by a spike and slab distribution, rather than a precise weight value. We then derive the mean-field algorithm to train the network at the ensemble level. The method is then applied to classify handwritten digits when pixels are read in sequence, and to the multisensory integration task that is a fundamental cognitive function of animals. Our model reveals important connections that determine the overall performance of the network. The model also shows how spatio-temporal information is processed through the hyperparameters of the distribution, and moreover reveals distinct types of emergent neural selectivity. It is thus promising to study the temporal credit assignment in recurrent neural networks from the ensemble perspective.
翻译:经常性神经网络被广泛用于在自然语言处理和神经人口动态中模拟时空序列。 但是, 理解时间信用分配是困难的。 在这里, 我们提议, 经常计算中的每个个体连接都用钉子和板块分布而不是精确的重量值来模拟。 然后我们得出平均场算法来在共同值一级培训网络。 然后, 在阅读像素时, 使用这种方法对手写的数字进行分类, 并且对作为动物基本认知功能的多感官整合任务进行分类。 我们的模型揭示了决定网络总体性能的重要连接。 该模型还显示, spatio- 时间信息是如何通过分布的超参数处理的, 并且还揭示了不同种类的突发神经选择性。 因此, 从共感角度研究经常性神经网络的时间信用分配, 很有希望从共感应变的角度看问题。