We study the excess capacity of deep networks in the context of supervised classification. That is, given a capacity measure of the underlying hypothesis class - in our case, empirical Rademacher complexity - to what extent can we (a priori) constrain this class while retaining an empirical error on a par with the unconstrained regime? To assess excess capacity in modern architectures (such as residual networks), we extend and unify prior Rademacher complexity bounds to accommodate function composition and addition, as well as the structure of convolutions. The capacity-driving terms in our bounds are the Lipschitz constants of the layers and an (2, 1) group norm distance to the initializations of the convolution weights. Experiments on benchmark datasets of varying task difficulty indicate that (1) there is a substantial amount of excess capacity per task, and (2) capacity can be kept at a surprisingly similar level across tasks. Overall, this suggests a notion of compressibility with respect to weight norms, complementary to classic compression via weight pruning. Source code is available at https://github.com/rkwitt/excess_capacity.
翻译:我们从监督分类的角度研究深层网络的过剩能力,也就是说,根据对基本假设等级的能力量度 -- -- 就我们而言,经验型雷德马赫复杂程度 -- -- 我们(先验性)能够在多大程度上约束这一类别,同时保留与不受限制的制度相同的经验错误?为了评估现代建筑(如残余网络)的过剩能力,我们扩展和统一了先前的雷德马赫复杂的界限,以适应功能构成和增加,以及组合结构。我们的界限中能力驱动术语是层层的利普西茨常数和(2,1)组标准距离共振量重量初始化的距离。任务难度不同的基准数据集实验表明:(1) 每项任务有大量的超载能力,(2) 能力可以保持在惊人的类似水平。总体而言,这表明了与重量规范的可压缩性概念,通过重量调整对经典压缩进行补充。源码见https://github.com/rkwitt/excess_capit。