The performance of machine learning models under distribution shift has been the focus of the community in recent years. Most of current methods have been proposed to improve the robustness to distribution shift from the algorithmic perspective, i.e., designing better training algorithms to help the generalization in shifted test distributions. This paper studies the distribution shift problem from the perspective of pre-training and data augmentation, two important factors in the practice of deep learning that have not been systematically investigated by existing work. By evaluating seven pre-trained models, including ResNets and ViT's with self-supervision and supervision mode, on five important distribution-shift datasets, from WILDS and DomainBed benchmarks, with five different learning algorithms, we provide the first comprehensive empirical study focusing on pre-training and data augmentation. With our empirical result obtained from 1,330 models, we provide the following main observations: 1) ERM combined with data augmentation can achieve state-of-the-art performance if we choose a proper pre-trained model respecting the data property; 2) specialized algorithms further improve the robustness on top of ERM when handling a specific type of distribution shift, e.g., GroupDRO for spurious correlation and CORAL for large-scale out-of-distribution data; 3) Comparing different pre-training modes, architectures and data sizes, we provide novel observations about pre-training on distribution shift, which sheds light on designing or selecting pre-training strategy for different kinds of distribution shifts. In summary, our empirical study provides a comprehensive baseline for a wide range of pre-training models fine-tuned with data augmentation, which potentially inspires research exploiting the power of pre-training and data augmentation in the future of distribution shift study.
翻译:近些年来,分布式转变中的机器学习模型的绩效一直是社区的重点。目前大多数方法都是为了从算法角度,即设计更好的培训算法,帮助在转换测试分布时进行概括化。本文从培训前和数据扩充的角度研究分配变化问题,这是现有工作尚未系统调查的深层次学习做法的两个重要因素。通过评估7个预先培训模式,包括ResNets和VIT以自我监督性能和监督模式的自控性变动,从WILDS和DomaineBed综合基准的5个重要的分布式基本数据集,从5个不同的学习算法,我们提供了第一个全面的经验性研究,重点是培训前和数据扩增。我们从1,330个模型获得的经验性结果,我们提供了以下主要观察:(1) 机构化与数据扩增前工作相结合,如果我们选择一个尊重数据属性的经过事先训练的模型,则可以产生自我监督的自我监督的自我风险管理顶部的自我监督性功能,从WILLDS和Dmaild Breal-real-real-real-reaction a commodistrational retrading laveal resulational lautes lauts the wes laveal laveal