Selective classification is the task of rejecting inputs a model would predict incorrectly on through a trade-off between input space coverage and model accuracy. Current methods for selective classification impose constraints on either the model architecture or the loss function; this inhibits their usage in practice. In contrast to prior work, we show that state-of-the-art selective classification performance can be attained solely from studying the (discretized) training dynamics of a model. We propose a general framework that, for a given test input, monitors metrics capturing the disagreement with the final predicted label over intermediate models obtained during training; we then reject data points exhibiting too much disagreement at late stages in training. In particular, we instantiate a method that tracks when the label predicted during training stops disagreeing with the final predicted label. Our experimental evaluation shows that our method achieves state-of-the-art accuracy/coverage trade-offs on typical selective classification benchmarks. For example, we improve coverage on CIFAR-10/SVHN by 10.1%/1.5% respectively at a fixed target error of 0.5%.
翻译:选择性分类是拒绝输入的任务,一个模型通过输入空间覆盖面和模型准确性之间的权衡而预测错误。目前选择性分类的方法对模型结构或损失功能施加限制;这在实践中抑制了它们的使用。与先前的工作不同,我们显示,最先进的选择性分类性能只能通过研究一个模型的(分解的)培训动态才能达到。我们提议了一个总框架,对特定测试投入而言,用来监测与最终预测的标签在培训期间获得的中间模型的不一致程度;然后我们拒绝在培训的后期阶段表现出太多分歧的数据点。特别是,当培训期间预测的标签不再与最后预测的标签不一致时,我们即刻使用一种方法。我们的实验性评估表明,我们的方法在典型的选择性分类基准上达到了最新水平的准确/覆盖交易。例如,我们把CFAR-10/SVHN的覆盖范围分别提高10.1%/1.5%,固定目标误差为0.5%。