Reliable probability estimation is of crucial importance in many real-world applications where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles. Probability-estimation models are trained on observed outcomes (e.g. whether it has rained or not, or whether a patient has died or not), because the ground-truth probabilities of the events of interest are typically unknown. The problem is therefore analogous to binary classification, with the important difference that the objective is to estimate probabilities rather than predicting the specific outcome. The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. There exist several methods to improve the probabilities generated by these models but they mostly focus on classification problems where the probabilities are related to model uncertainty. In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground-truth probabilities. To address this, we build a synthetic dataset to study and compare different computable metrics. We evaluate existing methods on the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty. We also give a theoretical analysis of a model for high-dimensional probability estimation which reproduces several of the phenomena evinced in our experiments. Finally, we propose a new method for probability estimation using neural networks, which modifies the training process to promote output probabilities that are consistent with empirical probabilities computed from the data. The method outperforms existing approaches on most metrics on the simulated as well as real-world data.
翻译:可靠的概率估算对于许多现实世界应用中存在内在不确定性的地方至关重要,如天气预报、医学预测或自动车辆避免碰撞等。 概率估算模型在观测结果方面受过培训(例如,是否下雨,病人是否死亡,或病人是否死亡),因为与感兴趣的事件的地面概率通常是未知的。因此,问题与二进制分类相似,其重要区别在于,目标是估算概率而不是预测具体结果。这项工作的目标是利用深层神经网络调查高维数据的概率估算。存在几种方法改进这些模型产生的概率,但它们主要侧重于与模型不确定性相关的分类问题。在存在内在不确定性的情况下,评估业绩时将面临挑战,而没有获得地面概率的概率。为了解决这一问题,我们建立一个合成数据集,以研究和比较不同的可比较指标。我们评估现有关于合成数据的现有方法,即利用最深的线性数据估算方法,在三个真实世界的概率估算中,我们用一个不断的精确的精确度数据模型来提出一个真实的概率估算。