Large-scale deep neural networks consume expensive training costs, but the training results in less-interpretable weight matrices constructing the networks. Here, we propose a mode decomposition learning that can interpret the weight matrices as a hierarchy of latent modes. These modes are akin to patterns in physics studies of memory networks. The mode decomposition learning not only saves a significant large amount of training costs, but also explains the network performance with the leading modes. The mode learning scheme shows a progressively compact latent space across the network hierarchy, and the least number of modes increases only logarithmically with the network width. Our mode decomposition learning is also studied in an analytic on-line learning setting, which reveals multi-stage of learning dynamics. Therefore, the proposed mode decomposition learning points to a cheap and interpretable route towards the magical deep learning.
翻译:大型深层神经网络耗资昂贵的培训费用,但培训结果导致建设网络的重量矩阵解释得较少。 在这里, 我们提出一种模式分解学习方法, 可以将重量矩阵解释为潜伏模式的等级。 这些模式类似于记忆网络物理研究的模式。 模式分解学习不仅节省了大量的培训费用, 而且还解释了主要模式的网络性能。 模式学习方案显示整个网络等级的潜伏空间逐渐紧凑, 模式最少的只是随着网络宽度的对数增加。 我们的模式分解学习也在一个分析式的在线学习环境中进行, 它揭示了学习动态的多阶段。 因此, 拟议的模式分解方法将学习指向一种廉价和可解释的走向神奇深层次学习的道路。