Numerical experiments demonstrate that deep neural network classifiers progressively separate class distributions around their mean, achieving linear separability on the training set, and increasing the Fisher discriminant ratio. We explain this mechanism with two types of operators. We prove that a rectifier without biases applied to sign-invariant tight frames can separate class means and increase Fisher ratios. On the opposite, a soft-thresholding on tight frames can reduce within-class variabilities while preserving class means. Variance reduction bounds are proved for Gaussian mixture models. For image classification, we show that separation of class means can be achieved with rectified wavelet tight frames that are not learned. It defines a scattering transform. Learning $1 \times 1$ convolutional tight frames along scattering channels and applying a soft-thresholding reduces within-class variabilities. The resulting scattering network reaches the classification accuracy of ResNet-18 on CIFAR-10 and ImageNet, with fewer layers and no learned biases.
翻译:数字实验表明,深神经网络分类者逐渐将班级分布围绕其平均值分开,在训练组上实现线性分离,并增加渔民差异比。我们用两种操作者来解释这个机制。我们证明,对标志-变化型紧身框架不加偏差地应用的校正装置可以将班级区分开来,并增加渔民比率。相反,紧身架上的软悬浮装置可以减少班级内部的变异性,同时保留班级手段。Gaussian混合模型的差别减少界限得到了证明。关于图像分类,我们显示,班级手段的分离可以用纠正的波盘紧身框架来实现,而这种框架是没有学到的。它定义了散射变。在散射通道上学习了1美元乘以连动性紧身框架,并应用软伸缩式紧身来减少班级变异性。由此形成的散网达到CIFAR-10和图像网ResNet-18的分类精度,其层次和不见识的偏差。