We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI). Contrary to other CMI bounds, which are black-box bounds that do not exploit the structure of the problem and may be hard to evaluate in practice, our loo-CMI bounds can be computed easily and can be interpreted in connection to other notions such as classical leave-one-out cross-validation, stability of the optimization algorithm, and the geometry of the loss-landscape. It applies both to the output of training algorithms as well as their predictions. We empirically validate the quality of the bound by evaluating its predicted generalization gap in scenarios for deep learning. In particular, our bounds are non-vacuous on large-scale image-classification tasks.
翻译:我们根据“放假一出”有条件相互信息(Loo-CMI)的新措施,为受监督的学习算法(Loo-CMI)得出信息理论概括界限。 与其他“CMI”界限相反,这些界限是黑箱界限,没有利用问题的结构,在实践中可能难以评估,我们“Loo-CMI”界限可以很容易地计算,并且可以结合其他概念解释,如传统的“放假一出”交叉验证、优化算法的稳定性和损失地貌的几何学,既适用于培训算法的产出,也适用于其预测。我们通过评估其预测的深层次学习情景中的“放学”差距,对约束的质量进行了实证,特别是,我们的界限在大规模图像分类任务上是无懈可击的。