Despite superior performance in many situations, deep neural networks are often vulnerable to adversarial examples and distribution shifts, limiting model generalization ability in real-world applications. To alleviate these problems, recent approaches leverage distributional robustness optimization (DRO) to find the most challenging distribution, and then minimize loss function over this most challenging distribution. Regardless of achieving some improvements, these DRO approaches have some obvious limitations. First, they purely focus on local regularization to strengthen model robustness, missing a global regularization effect which is useful in many real-world applications (e.g., domain adaptation, domain generalization, and adversarial machine learning). Second, the loss functions in the existing DRO approaches operate in only the most challenging distribution, hence decouple with the original distribution, leading to a restrictive modeling capability. In this paper, we propose a novel regularization technique, following the veins of Wasserstein-based DRO framework. Specifically, we define a particular joint distribution and Wasserstein-based uncertainty, allowing us to couple the original and most challenging distributions for enhancing modeling capability and applying both local and global regularizations. Empirical studies on different learning problems demonstrate that our proposed approach significantly outperforms the existing regularization approaches in various domains: semi-supervised learning, domain adaptation, domain generalization, and adversarial machine learning.
翻译:尽管在许多情况中表现优异,但深神经网络往往易受对抗性实例和分布变化的影响,限制了现实世界应用中的典型普及能力。为了缓解这些问题,最近的做法利用分配稳健性优化(DRO)来寻找最具挑战性的分布,然后在最具挑战性的分布中最大限度地减少损失功能。尽管取得了一些改进,但这些DRO方法还是有一些明显的局限性。首先,它们完全侧重于地方规范,以加强模型稳健性,缺乏全球规范化效应,而这种效应在许多现实世界应用(例如域适应、域一般化和对抗性机器学习)中是有用的。第二,现有DRO方法的损失功能只在最具挑战性的分布中运作,因此与最初的分布脱钩,从而导致一种限制性的建模能力。在本文件中,我们提出了一种新型的规范化方法,遵循了瓦塞斯坦德罗斯坦的DRO框架。具体地说,我们定义了一种特殊的联合分布和基于瓦塞斯坦的不确定性,使我们能够将原始和最具挑战性的分布结合起来,用于加强建模能力以及应用当地和全球规范化。第二,现有DRO方法的丧失功能。关于不同学习模式化的原始和实地研究领域的研究方法表明我们拟议的不同领域的正规化方法。