While deep learning through empirical risk minimization (ERM) has succeeded at achieving human-level performance at a variety of complex tasks, ERM is not robust to distribution shifts or adversarial attacks. Synthetic data augmentation followed by empirical risk minimization (DA-ERM) is a simple and widely used solution to improve robustness in ERM. In addition, consistency regularization can be applied to further improve the robustness of the model by forcing the representation of the original sample and the augmented one to be similar. However, existing consistency regularization methods are not applicable to covariant data augmentation, where the label in the augmented sample is dependent on the augmentation function. For example, dialog state covaries with named entity when we augment data with a new named entity. In this paper, we propose data augmented loss invariant regularization (DAIR), a simple form of consistency regularization that is applied directly at the loss level rather than intermediate features, making it widely applicable to both invariant and covariant data augmentation regardless of network architecture, problem setup, and task. We apply DAIR to real-world learning problems involving covariant data augmentation: robust neural task-oriented dialog state tracking and robust visual question answering. We also apply DAIR to tasks involving invariant data augmentation: robust regression, robust classification against adversarial attacks, and robust ImageNet classification under distribution shift. Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal computational cost and sets new state-of-the-art results in several benchmarks involving covariant data augmentation. Our code of all experiments is available at: https://github.com/optimization-for-data-driven-science/DAIR.git
翻译:虽然通过实验风险最小化(ERM)的深层学习成功地在各种复杂任务中实现了人类层面的绩效,但机构风险管理对于分布变化或对抗性攻击并不健全。合成数据增强,然后实验风险最小化(DA-ERM)是一个简单和广泛使用的解决办法,可以提高机构风险管理的稳健性。此外,一致性规范化可以用来通过强迫原始样本的表示形式和扩充的类似功能来进一步提高模型的稳健性。然而,现有的一致性规范化方法不适用于共变数据增强,而增加样本中的标签取决于增强功能。例如,当我们与一个新命名实体增加数据时,与指定实体对话状态的covery。在本文件中,我们提议的数据会增加差异性规范化(DAIR),这是一种简单的一致性规范化形式,直接适用于损失水平而不是中间特征,使得它广泛适用于差异性和变量性数据增强,而不论网络结构、问题设置和任务如何。我们应用DAIR的实时学习结果,涉及可变性数据增强性数据增强:以坚固的内向性任务定位的状态对准性(OADAAA-AAAAAAAA级)的精确性数据分类的跟踪和动态数据升级分析。