Deep-predictive-coding networks (DPCNs) are hierarchical, generative models that rely on feed-forward and feed-back connections to modulate latent feature representations of stimuli in a dynamic and context-sensitive manner. A crucial element of DPCNs is a forward-backward inference procedure to uncover sparse states of a dynamic model, which are used for invariant feature extraction. However, this inference and the corresponding backwards network parameter updating are major computational bottlenecks. They severely limit the network depths that can be reasonably implemented and easily trained. We therefore propose a optimization strategy, with better empirical and theoretical convergence, based on accelerated proximal gradients. We demonstrate that the ability to construct deeper DPCNs leads to receptive fields that capture well the entire notions of objects on which the networks are trained. This improves the feature representations. It yields completely unsupervised classifiers that surpass convolutional and convolutional-recurrent autoencoders and are on par with convolutional networks trained in a supervised manner. This is despite the DPCNs having orders of magnitude fewer parameters.
翻译:深度预知编码网络(DPCN)是等级、基因化模型,依赖进料前和回馈连接,以动态和背景敏感的方式调节刺激的潜在特征表现。DPCN的一个关键要素是前向后推推推程序,以发现动态模型的稀疏状态,这种动态模型用于异变特征提取。然而,这种推论和相应的逆向网络参数更新是主要的计算瓶颈,严重限制了可以合理实施和容易培训的网络深度。因此,我们提出了一个优化战略,以加速的准氧化梯度为基础,在经验和理论上更好地融合。我们表明,更深的DPCN能够导致接收领域,捕捉到网络所培训对象的全部概念。这改进了特征表达方式。它产生完全不受监督的分类器,超越了革命和革命的经常自动电算器,与以监督方式培训的革命网络是相同的。尽管DPCN的参数数量级较低。