We define a notion of complexity, which quantifies the nonlinearity of the computation of a neural network, as well as a complementary measure of the effective dimension of feature representations. We investigate these observables both for trained networks for various datasets as well as explore their dynamics during training, uncovering in particular power law scaling. These observables can be understood in a dual way as uncovering hidden internal structure of the datasets themselves as a function of scale or depth. The entropic character of the proposed notion of complexity should allow to transfer modes of analysis from neuroscience and statistical physics to the domain of artificial neural networks. The introduced observables can be applied without any change to the analysis of biological neuronal systems.
翻译:我们定义了一个复杂的概念,它量化了神经网络计算的非线性,以及特征表现的有效层面的补充度量。我们调查了这些可观测到的情况,既用于各种数据集的训练有素的网络,也用于在培训期间探索其动态,特别揭示了权力法的尺度。这些可观测到的情况可以被理解为揭示数据集本身隐藏的内部结构是规模或深度的函数。提议的复杂概念的进化特性应允许将分析模式从神经科学和统计物理学转移到人造神经网络领域。引入的可观测可以不作任何改变地应用到生物神经系统的分析中。