Quantifying which neurons are important with respect to the classification decision of a trained neural network is essential for understanding their inner workings. Previous work primarily attributed importance to individual neurons. In this work, we study which groups of neurons contain synergistic or redundant information using a multivariate mutual information method called the O-information. We observe the first layer is dominated by redundancy suggesting general shared features (i.e. detecting edges) while the last layer is dominated by synergy indicating local class-specific features (i.e. concepts). Finally, we show the O-information can be used for multi-neuron importance. This can be demonstrated by re-training a synergistic sub-network, which results in a minimal change in performance. These results suggest our method can be used for pruning and unsupervised representation learning.
翻译:在经过培训的神经网络的分类决定方面,对哪些神经元很重要进行量化,对于了解其内部工作至关重要。以前的工作主要与个别神经元有关。在这项工作中,我们研究哪些神经元组含有协同或冗余信息,使用称为O-信息的一种多变的相互信息方法。我们观察第一层的特点是,存在一些冗余,表明一般共有特征(即探测边缘),而最后一个层则以显示当地类别特征(即概念)的协同作用为主。最后,我们显示O-信息可用于多中子的重要性。这可以通过对协同子网络进行再培训来证明,从而导致最低限度的性能变化。这些结果表明,我们的方法可以用于剪裁和无监督的代言学习。