Interpretability of Deep Neural Networks has become a major area of exploration. Although these networks have achieved state of the art accuracy in many tasks, it is extremely difficult to interpret and explain their decisions. In this work we analyze the final and penultimate layers of Deep Convolutional Networks and provide an efficient method for identifying subsets of features that contribute most towards the network's decision for a class. We demonstrate that the number of such features per class is much lower in comparison to the dimension of the final layer and therefore the decision surface of Deep CNNs lies on a low dimensional manifold and is proportional to the network depth. Our methods allow to decompose the final layer into separate subspaces which is far more interpretable and has a lower computational cost as compared to the final layer of the full network.
翻译:深神经网络的可解释性已成为一个重要的勘探领域。 虽然这些网络在许多任务中达到了最新水平的准确性,但解释和解释其决定极为困难。 在这项工作中,我们分析了深革命网络的最后层和倒数倒数第二层,并提供了一个有效的方法,用以确定最有助于网络就某一类作出决定的特征子集。我们证明,与最后一层相比,每层这类特征的数量要低得多,因此深CNN的判定面位于一个低维的多元体上,并且与网络的深度成正比。我们的方法可以将最后一层分解成不同的子空间,而这些子空间的解释性要大得多,而且与整个网络的最后一层相比,计算成本也较低。