With the proliferation of mobile devices and the Internet of Things, deep learning models are increasingly deployed on devices with limited computing resources and memory, and are exposed to the threat of adversarial noise. Learning deep models with both lightweight and robustness is necessary for these equipments. However, current deep learning solutions are difficult to learn a model that possesses these two properties without degrading one or the other. As is well known, the fully-connected layers contribute most of the parameters of convolutional neural networks. We perform a separable structural transformation of the fully-connected layer to reduce the parameters, where the large-scale weight matrix of the fully-connected layer is decoupled by the tensor product of several separable small-sized matrices. Note that data, such as images, no longer need to be flattened before being fed to the fully-connected layer, retaining the valuable spatial geometric information of the data. Moreover, in order to further enhance both lightweight and robustness, we propose a joint constraint of sparsity and differentiable condition number, which is imposed on these separable matrices. We evaluate the proposed approach on MLP, VGG-16 and Vision Transformer. The experimental results on datasets such as ImageNet, SVHN, CIFAR-100 and CIFAR10 show that we successfully reduce the amount of network parameters by 90%, while the robust accuracy loss is less than 1.5%, which is better than the SOTA methods based on the original fully-connected layer. Interestingly, it can achieve an overwhelming advantage even at a high compression rate, e.g., 200 times.
翻译:随着移动装置和事物互联网的扩散,深度学习模型越来越多地部署在计算机资源和记忆有限的设备上,并暴露在对抗性噪音的威胁之下。学习具有轻量和强力的深模型对于这些设备是必要的。然而,目前深层学习的解决方案很难学习一个拥有这两种特性而不降低其中一种或另一种特性的模型。众所周知,完全连接的层层对动态神经网络的大部分参数都有帮助作用。我们对完全连接的层进行分解的结构转型,以减少参数,完全连接层的大规模重量矩阵被若干可分解的小型矩阵的高压产品分解。我们评估了MLP、VGG-16等数据在被完全连接的层之前不再需要被粉碎,保留了宝贵的空间测量数据。此外,为了进一步增强轻度和强度,我们建议对完全连接的层进行联合限制,对完全连接的层结构设置不同的条件。我们评估了MLP、NGFAR-16的大规模重量矩阵的分流数据,而SFAR10的精确度则降低了SAR10的精确度。我们用SFAR10的精确度模型和视觉变压数据在SAR10的精确度上得出了更好的数字。