Reliably estimating the uncertainty of a prediction throughout the model lifecycle is crucial in many safety-critical applications. The most common way to measure this uncertainty is via the predicted confidence. While this tends to work well for in-domain samples, these estimates are unreliable under domain drift and restricted to classification. Alternatively, proper scores can be used for most predictive tasks but a bias-variance decomposition for model uncertainty does not exist in the current literature. In this work we introduce a general bias-variance decomposition for proper scores, giving rise to the Bregman Information as the variance term. We discover how exponential families and the classification log-likelihood are special cases and provide novel formulations. Surprisingly, we can express the classification case purely in the logit space. We showcase the practical relevance of this decomposition on several downstream tasks, including model ensembles and confidence regions. Further, we demonstrate how different approximations of the instance-level Bregman Information allow reliable out-of-distribution detection for all degrees of domain drift.
翻译:在许多安全关键应用中,可靠地估计模型生命周期内的预测不确定性至关重要。衡量此不确定性最常用的方法是通过预测的置信度。尽管这种方法在样本内表现良好,在域漂移下这些估计是不可靠的,并且仅限于分类。作为替代,可以对大多数预测任务使用适当的分数,但目前文献中不存在用于模型不确定性的偏差-方差分解。在这项工作中,我们引入了适用于适当分数的通用偏差-方差分解,从而产生了Bregman信息作为方差项。我们发现指数族和分类对数似然是特殊情况,并提供了新的公式。令人惊奇的是,我们可以在logit空间中纯粹地表达分类案例。我们展示了这种分解在几个下游任务中的实际相关性,包括模型集合和置信区域。此外,我们展示了不同的实例级别Bregman信息的近似方法如何使得可以可靠地检测所有程度的域漂移。