Though the background is an important signal for image classification, over reliance on it can lead to incorrect predictions when spurious correlations between foreground and background are broken at test time. Training on a dataset where these correlations are unbiased would lead to more robust models. In this paper, we propose such a dataset called Diffusion Dreamed Distribution Shifts (D3S). D3S consists of synthetic images generated through StableDiffusion using text prompts and image guides obtained by pasting a sample foreground image onto a background template image. Using this scalable approach we generate 120K images of objects from all 1000 ImageNet classes in 10 diverse backgrounds. Due to the incredible photorealism of the diffusion model, our images are much closer to natural images than previous synthetic datasets. D3S contains a validation set of more than 17K images whose labels are human-verified in an MTurk study. Using the validation set, we evaluate several popular DNN image classifiers and find that the classification performance of models generally suffers on our background diverse images. Next, we leverage the foreground & background labels in D3S to learn a foreground (background) representation that is invariant to changes in background (foreground) by penalizing the mutual information between the foreground (background) features and the background (foreground) labels. Linear classifiers trained on these features to predict foreground (background) from foreground (background) have high accuracies at 82.9% (93.8%), while classifiers that predict these labels from background and foreground have a much lower accuracy of 2.4% and 45.6% respectively. This suggests that our foreground and background features are well disentangled. We further test the efficacy of these representations by training classifiers on a task with strong spurious correlations.
翻译:虽然背景是图像分类的一个重要信号, 但对于它的依赖可能会导致不正确的预测, 当测试时间折断了前景和背景之间表面之间虚假的关联时, 当测试时间打破了测试背景时, 我们可能会导致不正确的预测。 在这种关联没有偏差的数据集上的培训会导致更强的模型。 在本文中, 我们提出这样的数据集, 叫做 Difultion Dream 分布移位( D3S ) 。 D3S 是由通过 StailDifulation 生成的合成图像构成的, 使用文本提示和图像指南在背景模板图像上粘贴一个样本的地面图像。 使用这种可缩放的方法, 我们以10种不同的背景, 从所有 1000 下层图像网络类中生成 120K 对象的图像。 由于扩散模型具有令人难以置信的摄影真实性, 我们的图像比以前的合成数据集更接近自然图像。 D3S 包含一套超过 17K 图像的验证集集, 其标签在 MTurk 任务研究中被人类验证。 我们用一些流行的 DNN 图像分类分析器进一步( ) 并发现模型的分类表现在背景图层图像上, 从 D3Sloorthroder lader lader lader lader lader 和 lader lader 。