Important research efforts have focused on the design and training of neural networks with a controlled Lipschitz constant. The goal is to increase and sometimes guarantee the robustness against adversarial attacks. Recent promising techniques draw inspirations from different backgrounds to design 1-Lipschitz neural networks, just to name a few: convex potential layers derive from the discretization of continuous dynamical systems, Almost-Orthogonal-Layer proposes a tailored method for matrix rescaling. However, it is today important to consider the recent and promising contributions in the field under a common theoretical lens to better design new and improved layers. This paper introduces a novel algebraic perspective unifying various types of 1-Lipschitz neural networks, including the ones previously mentioned, along with methods based on orthogonality and spectral methods. Interestingly, we show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. We also prove that AOL biases the scaled weight to the ones which are close to the set of orthogonal matrices in a certain mathematical manner. Moreover, our algebraic condition, combined with the Gershgorin circle theorem, readily leads to new and diverse parameterizations for 1-Lipschitz network layers. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers. Finally, the comprehensive set of experiments on image classification shows that SLLs outperform previous approaches on certified robust accuracy. Code is available at https://github.com/araujoalexandre/Lipschitz-SLL-Networks.
翻译:重要的研究努力集中在设计和培训神经网络,并有一个受控制的利普申茨常数。目标是增加有时保证对抗性攻击的稳健性。最近有希望的技术从不同背景中汲取灵感,设计1-利普申茨神经网络,仅举几个例子:convex潜在层来自连续动态系统的离散,几乎Orthogonal-Layer提出一个量身定制的矩阵调整方法。然而,今天重要的是,在共同的理论角度下考虑最近和有希望的实地贡献,以便更好地设计新的和改进的层次。本文介绍了一种新型的变校正视角,统一了各种类型的1-利普申神经网络的准确性,包括以前提到的那些网络。有趣的是,我们现有的许多技术可以通过找到分析方法来产生和普及,通过找到一个共同的半变性编程程序(SDP)的条件。我们还证明,AOL偏重重到接近或经改进的立体矩阵的组合,以某种数学方式。此外,我们的algebralial-LIRS 模型显示我们整个结构的变形结构。Sal-ligreal-liversal-listration-Ls</s>