A learning method is self-certified if it uses all available data to simultaneously learn a predictor and certify its quality with a statistical certificate that is valid on unseen data. Recent work has shown that neural network models trained by optimising PAC-Bayes bounds lead not only to accurate predictors, but also to tight risk certificates, bearing promise towards achieving self-certified learning. In this context, learning and certification strategies based on PAC-Bayes bounds are especially attractive due to their ability to leverage all data to learn a posterior and simultaneously certify its risk. In this paper, we assess the progress towards self-certification in probabilistic neural networks learnt by PAC-Bayes inspired objectives. We empirically compare (on 4 classification datasets) classical test set bounds for deterministic predictors and a PAC-Bayes bound for randomised self-certified predictors. We first show that both of these generalisation bounds are not too far from out-of-sample test set errors. We then show that in data starvation regimes, holding out data for the test set bounds adversely affects generalisation performance, while self-certified strategies based on PAC-Bayes bounds do not suffer from this drawback, proving that they might be a suitable choice for the small data regime. We also find that probabilistic neural networks learnt by PAC-Bayes inspired objectives lead to certificates that can be surprisingly competitive with commonly used test set bounds.
翻译:如果使用所有可用数据同时学习预测器并用对不可见数据有效的统计证书核证其质量,一种学习方法是自我认证的。最近的工作表明,通过优化 PAC-Bayes 边框来优化PAC-Bayes 边框培训的神经网络模型不仅导致准确的预测器,而且会导致严格的风险证书,从而有可能实现自我认证的学习。在这方面,基于PAC-Bayes 边框的学习和认证战略特别具有吸引力,因为它们能够利用所有数据学习外表,同时验证其风险。在本文中,我们评估了PAC-Bayes 启发的目标所学的概率神经网络自我认证的进展情况。我们用经验比较(在4个分类数据集中) 典型的测试框定了确定预测器,而PAC-Bayes 边框定的典型测试框定框框框框框框框不仅会影响常规的神经神经网络的常规测试目标,我们首先可以发现在数据饥饿制度中保留用于测试的测试框框框框框框框框架的常规测试策略。