Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
翻译:选择性分类(或有拒绝选项的分类)对一个具有选择功能的分类员,以确定是否接受预测。这个框架将覆盖面(接受预测的可能性)与预测性表现(通常以分配性损失功能衡量)相权衡。在许多应用情景中,例如信用评分,业绩则用等级衡量,如ROC曲线下的区域(AUC),我们建议一种模式性、不可知性的办法,将选择功能与特定概率性二元分类员挂钩。这个办法具体针对优化ACC。我们既提供了理论理由,又提供了新奇的算法,称为AUCROSS,以实现这一目标。实验表明,我们的方法成功地实现了ACUC的交易性覆盖,改进了旨在优化准确性的现有选择性分类方法。