We study methods for estimating model uncertainty for neural networks (NNs) in regression. To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for NNs, which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-NNs, one for model prediction and one for model uncertainty, and to train it using a carefully-designed loss function. Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture, NOMU can provide model uncertainty for any given (previously trained) NN if given access to its training data. We evaluate NOMU in various regressions tasks and noiseless Bayesian optimization (BO) with costly evaluations. In regression, NOMU performs at least as well as state-of-the-art methods. In BO, NOMU even outperforms all considered benchmarks.
翻译:研究神经网络在回归中的模型不确定性的方法。为了分离模型不确定性的影响,我们侧重于无噪音的模型不确定性。我们引入了5种关于任何方法都应满足的模型不确定性的重要偏差。然而,我们发现,既定基准往往没有可靠地捕捉到其中一些偏差,甚至是巴伊西亚理论所要求的偏差。为了解决这个问题,我们引入了一种新的方法来捕捉NNN的模型不确定性,我们称之为以神经优化为基础的模型不确定性。NOMU的主要想法是设计一个由两个连接的子NNW组成的网络结构,一个用于模型预测,一个用于模型不确定性,并使用精心设计的损失函数来培训它。重要的是,我们的设计使NOMU满足了我们五个偏差功能。由于它的模块结构,NOMU可以提供模型不确定性,任何给NNN(以前受过训练的)培训数据,我们称之为NNMU,我们在各种回归任务和无噪音海湾优化(BO)中评估各种费用高昂的任务。在回归中,NOMU至少以状态方式执行BOMU的状态。