Learning in the brain is poorly understood and learning rules that respect biological constraints, yet yield deep hierarchical representations, are still unknown. Here, we propose a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. Learning minimizes a simple layer-specific loss function and does not need to back-propagate error signals within or between layers. Instead, weight updates follow a local, Hebbian, learning rule that only depends on pre- and post-synaptic neuronal activity, predictive dendritic input and widely broadcasted modulation factors which are identical for large groups of neurons. The learning rule applies contrastive predictive learning to a causal, biological setting using saccades (i.e. rapid shifts in gaze direction). We find that networks trained with this self-supervised and local rule build deep hierarchical representations of images, speech and video.
翻译:大脑的学习不易理解,而学习规则尊重生物限制,但产生深层次的等级代表,仍然未知。在这里,我们提出一个学习规则,从神经科学和自我监督的深层学习的最新进展中汲取灵感。学习最大限度地减少了简单的层次损失功能,不需要在层内或层间反向传播错误信号。相反,体重更新遵循本地的Hebbian的学习规则,该规则仅取决于合成前和后神经活动、预测的登层输入和广泛广播的调节因素,这些因素对大群神经人来说是相同的。学习规则将对比式预测学习运用于一个因果生物环境,使用Scathdes(即快速转向瞄准方向 ) 。 我们发现,接受这种自我监管和本地规则培训的网络可以建立对图像、言语和视频的深层次描述。