This paper presents an empirical study regarding training probabilistic neural networks using training objectives derived from PAC-Bayes bounds. In the context of probabilistic neural networks, the output of training is a probability distribution over network weights. We present two training objectives, used here for the first time in connection with training neural networks. These two training objectives are derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned using the different training objectives. We compute risk certificates for the learnt predictors, based on part of the data used to learn the predictors. We further experiment with different types of priors on the weights (both data-free and data-dependent priors) and neural network architectures. Our experiments on MNIST and CIFAR-10 show that our training methods produce competitive test set errors and non-vacuous risk bounds with much tighter values than previous results in the literature, showing promise not only to guide the learning algorithm through bounding the risk but also for model selection. These observations suggest that the methods studied here might be good candidates for self-certified learning, in the sense of using the whole data set for learning a predictor and certifying its risk on any unseen data (from the same distribution as the training data) potentially without the need for holding out test data.
翻译:本文介绍了关于使用PAC-Bayes 线上的培训目标进行培训概率神经网络的经验研究。在概率神经网络中,培训产出是网络重量的概率分布。我们在此首次介绍了两个培训目标,这是在培训神经网络中首次使用的。这两个培训目标来自紧凑的PAC-Bayes-Bayes 线上。我们还根据古典PAC-Bayes 线上的培训目标,重新实施以前使用的培训目标,以比较利用不同培训目标所学预测者的特性。我们根据用于学习预测者的数据的一部分,为所学预测者计算风险证书。我们进一步试验了不同种类的先前重量(数据无数据性和数据依赖性)和神经网络结构。我们对MNIST和CIFAR-10号的实验表明,我们的培训方法产生了竞争性的错误和不含糊的风险约束,其价值比文献中以往的要紧得多,表明我们承诺不仅通过约束风险来指导学习算法,而且还要从模型选择中指导学习。我们进一步试验前期的预测者,我们进一步试验了各种前期数据,以便进行可靠的数据学习。