We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance. To ensure that these shifts are plausible, we parameterize them in terms of interpretable changes in causal mechanisms of observed variables. This defines a parametric robustness set of plausible distributions and a corresponding worst-case loss. While the loss under an individual parametric shift can be estimated via reweighting techniques such as importance sampling, the resulting worst-case optimization problem is non-convex, and the estimate may suffer from large variance. For small shifts, however, we can construct a local second-order approximation to the loss under shift and cast the problem of finding a worst-case shift as a particular non-convex quadratic optimization problem, for which efficient algorithms are available. We demonstrate that this second-order approximation can be estimated directly for shifts in conditional exponential family models, and we bound the approximation error. We apply our approach to a computer vision task (classifying gender from images), revealing sensitivity to shifts in non-causal attributes.
翻译:我们给出了一种方法,以积极主动地确定导致模型性能巨大差异的分布上的小、可信的变化,从而导致模型性能的巨大差异。为了确保这些变化是可信的,我们用观察到变量因果机制的可解释性变化来参数化这些变化。这定义了一组可信的分布和相应的最坏损失的参数性强。虽然单项参数性变化之下的损失可以通过诸如重要性抽样等重估技术来估计,但由此产生的最坏情况优化问题是非隐形的,而且估计可能存在巨大的差异。但是,对于小变化,我们可以构建一个本地的对轮值损失的二阶近似,并造成找到最坏情况变化的问题,作为特定的非convex二次优化问题,对此可以提供有效的算法。我们证明,对于有条件的指数式家庭模型的转变,可以直接估计这种第二阶值的近似值,而且我们把近似错误捆绑在一起。我们的方法应用于计算机的视觉任务(将性别从图像中分类),显示对非因果属性的变化的敏感度。