Adversarial training and data augmentation with noise are widely adopted techniques to enhance the performance of neural networks. This paper investigates adversarial training and data augmentation with noise in the context of regularized regression in a reproducing kernel Hilbert space (RKHS). We establish the limiting formula for these techniques as the attack and noise size, as well as the regularization parameter, tend to zero. Based on this limiting formula, we analyze specific scenarios and demonstrate that, without appropriate regularization, these two methods may have larger generalization error and Lipschitz constant than standard kernel regression. However, by selecting the appropriate regularization parameter, these two methods can outperform standard kernel regression and achieve smaller generalization error and Lipschitz constant. These findings support the empirical observations that adversarial training can lead to overfitting, and appropriate regularization methods, such as early stopping, can alleviate this issue.
翻译:对抗性训练和带噪音的数据增强是提升神经网络性能的广泛采用的技术。本文在再现核希尔伯特空间正则化回归的背景下,研究了对抗性训练和带噪音的数据增强。我们建立了这些技术的极限公式,当攻击和噪音大小以及正则化参数趋近于零时,可予以证实。基于这个极限公式,我们分析了具体的场景,并表明,如果没有适当的正则化,这两种方法可能具有比标准核回归更大的泛化误差和Lipschitz常数。然而,通过选择适当的正则化参数,这两种方法可以优于标准核回归,实现较小的泛化误差和Lipschitz常数。这些发现支持实证观察:对抗性训练可能导致过拟合,而适当的正则化方法(例如提前停止)可以缓解这个问题。