Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. ATC outperforms previous methods across several model architectures, types of distribution shifts (e.g., due to synthetic corruptions, dataset reproduction, or novel subpopulations), and datasets (Wilds, ImageNet, Breeds, CIFAR, and MNIST). In our experiments, ATC estimates target performance $2$-$4\times$ more accurately than prior methods. We also explore the theoretical foundations of the problem, proving that, in general, identifying the accuracy is just as hard as identifying the optimal predictor and thus, the efficacy of any method rests upon (perhaps unstated) assumptions on the nature of the shift. Finally, analyzing our method on some toy distributions, we provide insights concerning when it works. Code is available at https://github.com/saurabhgarg1996/ATC_code/.
翻译:实际世界机器学习部署的特点是源(培训)与目标(测试)分布不匹配,可能导致性能下降。在这项工作中,我们只使用标签源数据和未标签目标数据调查预测目标领域准确性的方法。我们提出平均阈值信任(ATC),这是一个实用方法,可以学习模型信任的阈值,预测准确性作为模型信任超过该阈值的未标记例子的一小部分。ATC在几个模型结构、分配变化类型(例如合成腐败、数据集复制或新型子群落)和数据集(Wilds、图像网、Breeds、CIFAR和MIST)中优于以往方法的不匹配性。在我们的实验中,ATC估计的性能比先前方法精确2美元-4美元。我们还探讨问题的理论基础,证明总体而言,确定准确性与确定最佳预测器和因此,任何方法的效率都取决于(例如合成腐败、数据集复制或新型子群集),以及数据集(Wilds、图像网、CIFAR和MIST)的假设。最后,在我们现有法流流法的分布上提供了我们的理解。