Covariate shift has been shown to sharply degrade both predictive accuracy and the calibration of uncertainty estimates for deep learning models. This is worrying, because covariate shift is prevalent in a wide range of real world deployment settings. However, in this paper, we note that frequently there exists the potential to access small unlabeled batches of the shifted data just before prediction time. This interesting observation enables a simple but surprisingly effective method which we call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. Using this one line code change, we achieve state-of-the-art on recent covariate shift benchmarks and an mCE of 60.28\% on the challenging ImageNet-C dataset; to our knowledge, this is the best result for any model that does not incorporate additional data augmentation or modification of the training pipeline. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness (e.g. deep ensembles) and combining the two further improves performance. Our findings are supported by detailed measurements of the effect of this strategy on model behavior across rigorous ablations on various dataset modalities. However, the method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift, and is therefore worthy of additional study. We include links to the data in our figures to improve reproducibility, including a Python notebooks that can be run to easily modify our analysis at https://colab.research.google.com/drive/11N0wDZnMQQuLrRwRoumDCrhSaIhkqjof.
翻译:COVIG 转换显示, 大幅降低预测准确性和深度学习模型的不确定性估计值的校准。 这令人担心, 因为在现实世界部署的多种环境中, COVI变换是普遍存在的。 然而, 在本文中, 我们注意到, 在预测时间之前, 常常存在获取小批未贴标签的变换数据的可能性。 这种有趣的观察使得我们称之为预测- 时间批次正常化的简单但令人惊讶的有效方法, 这在COVI 转换中, 大大提高了模型准确性和校准值。 使用这一行代码的修改, 我们实现了最新的COVI 变换基准的艺术状态, 在具有挑战性的图像网- C数据集中, 出现了60.28 的 mCE; 但是, 据我们所知, 这是任何模型不包含额外数据增强或修改培训管道的最好结果。 我们的预测- 批次正常化方法为改进稳健性( 例如, 深度变现) 和两次研究的改进性能。 我们的研究结果得到详细测量这一战略对数值的影响的支持, 在精确的RVLI 中, 在各种数据模式下, 进行更精确的变换数据方法之下, 似乎包括更精确的变为累进。