Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.
翻译:在有监督的学习环境中,利用深神经网络来产生校准预测概率的多种技术已经出现,这些技术利用各种办法,从多个随机起点(深孔)的周期培训或培训中发现多种混合解决办法,然而,只有有限的工作量调查了围绕每一种不同解决办法(别种模式)探索当地区域是否有用。使用CIFAR-10数据集上三个众所周知的深层结构,我们评估了在Brier评分、准确度和预期校准错误方面探索权重空间的局部区域的若干简单方法。我们认为,巴伊西亚推理技术(变异推断和汉密尔顿·蒙特卡洛都适用于软式马克思输出层)以及利用Popima附近的随机梯度梯度梯度下行轨都是一样。我们为共性一致地改进性能添加了不同的模式,同时我们表明,这里考虑的简单模式勘探方法在不进行模式勘探的情况下,不会产生任何改进。