State-of-the-art machine learning models can be vulnerable to very small input perturbations that are adversarially constructed. Adversarial training is one of the most effective approaches to defend against such examples. We show that for linear regression problems, adversarial training can be formulated as a convex problem. This fact is then used to show that $\ell_\infty$-adversarial training produces sparse solutions and has many similarities to the lasso method. Similarly, $\ell_2$-adversarial training has similarities with ridge regression. We use a robust regression framework to analyze and understand these similarities and also point to some differences. Finally, we show how adversarial training behaves differently from other regularization methods when estimating overparameterized models (i.e., models with more parameters than datapoints). It minimizes a sum of three terms which regularizes the solution, but unlike lasso and ridge regression, it can sharply transition into an interpolation mode. We show that for sufficiently many features or sufficiently small regularization parameters, the learned model perfectly interpolates the training data while still exhibiting good out-of-sample performance.
翻译:最先进的机器学习模式很容易受到极小的输入干扰,而这种干扰是对抗性构建的。反向培训是对抗这类例子的最有效方法之一。我们显示,对于线性回归问题,对抗性培训可以形成一个锥形问题。这个事实被用来表明,$\ ell\ infty$-对抗性培训产生稀疏的解决方案,与拉索方法有许多相似之处。同样,$\ ell_2$-对抗性培训与山脊回归相似。我们使用一个强大的回归框架来分析和理解这些相似之处,并且指出一些差异。最后,我们展示了在估计超度参数比数据点更多的模型时,对抗性培训与其他正规化方法不同(即模型)的行为方式。它最大限度地减少了三个词的总和,这三个词使解决方案规范化,但与拉索和山脊回归法不同,它可以急剧转换为一种内流模式。我们发现,对于许多特征或足够小的正规化参数,我们所学过的模型完美地将培训数据与仍然展示出好的外表。