We review the current literature concerned with information plane analyses of neural network classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis of how the respective information quantities were estimated. Our survey suggests that compression visualized in information planes is not necessarily information-theoretic, but is rather often compatible with geometric compression of the latent representations. This insight gives the information plane a renewed justification. Aside from this, we shed light on the problem of estimating mutual information in deterministic neural networks and its consequences. Specifically, we argue that even in feed-forward neural networks the data processing inequality need not hold for estimates of mutual information. Similarly, while a fitting phase, in which the mutual information between the latent representation and the target increases, is necessary (but not sufficient) for good classification performance, depending on the specifics of mutual information estimation such a fitting phase need not be visible in the information plane.
翻译:我们审视了目前与神经网络分类师信息平面分析有关的文献。 基本的信息瓶颈理论和关于信息理论与一般化有因果关系的说法是有道理的,但经验证据却被认为既支持又相互矛盾。 我们审视了这一证据,并详细分析了如何估计信息数量。 我们的调查表明,信息平面中的压缩图像不一定是信息理论,但往往与潜在表达面的几何压缩相容。 这种洞察为信息平面提供了新的理由。 除此之外,我们揭示了在确定性神经网络及其后果中估计相互信息的问题。 具体地说,我们争论说,即使在进食型神经网络中,数据处理不平等也不需要维持对相互信息的估计。 同样,一个适当阶段,即潜在代表面与目标增加之间的相互信息对于良好的分类性能来说是必要的(但并不足够 ), 取决于相互信息估计的具体程度,这种适当阶段不需要在信息平面上看到。