Certifying the robustness of model performance under bounded data distribution shifts has recently attracted intensive interests under the umbrella of distributional robustness. However, existing techniques either make strong assumptions on the model class and loss functions that can be certified, such as smoothness expressed via Lipschitz continuity of gradients, or require to solve complex optimization problems. As a result, the wider application of these techniques is currently limited by its scalability and flexibility -- these techniques often do not scale to large-scale datasets with modern deep neural networks or cannot handle loss functions which may be non-smooth, such as the 0-1 loss. In this paper, we focus on the problem of certifying distributional robustness for black box models and bounded losses, without other assumptions. We propose a novel certification framework given bounded distance of mean and variance of two distributions. Our certification technique scales to ImageNet-scale datasets, complex models, and a diverse range of loss functions. We then focus on one specific application enabled by such scalability and flexibility, i.e., certifying out-of-domain generalization for large neural networks and loss functions such as accuracy and AUC. We experimentally validate our certification method on a number of datasets, ranging from ImageNet, where we provide the first non-vacuous certified out-of-domain generalization, to smaller classification tasks where we are able to compare with the state-of-the-art and show that our method performs considerably better.
翻译:在受约束的数据分布变化下,验证模型性能的稳健性最近在分布稳健性伞状下引起了强烈的兴趣;然而,现有技术要么对模型类别和损失功能的可靠度作出强有力的假设,这些功能可以证明,例如,通过Lipschitz梯度连续性表示的平稳度,或者需要解决复杂的优化问题。因此,目前这些技术的更广泛应用受到其可缩放性和灵活性的限制 -- -- 这些技术往往不至于扩大到具有现代深层神经网络的大规模数据集,或无法处理可能非mooood的损失功能,例如0-1损失。在本文件中,我们侧重于对黑盒模型和捆绑损失的分布性功能进行验证,而无需其他假设。我们提出了一个新的认证框架,以两种分布的平均值和差异的界限相近距离为界限。我们的认证技术尺度是图像网络规模数据集、复杂模型和各种损失功能。我们随后侧重于一种由于这种可缩放性和灵活性而得以实现的特定应用,例如0-1损失。我们侧重于对大型神经网络和封闭性损失的分布进行校准性概括性验证,我们从一个更精确性和实验性模型的功能到一个更精确性测试性的方法。