We present a framework for ranking images within their class based on the strength of spurious cues present. By measuring the gap in accuracy on the highest and lowest ranked images (we call this spurious gap), we assess spurious feature reliance for $89$ diverse ImageNet models, finding that even the best models underperform in images with weak spurious presence. However, the effect of spurious cues varies far more dramatically across classes, emphasizing the crucial, often overlooked, class-dependence of the spurious correlation problem. While most spurious features we observe are clarifying (i.e. improving test-time accuracy when present, as is typically expected), we surprisingly find many cases of confusing spurious features, where models perform better when they are absent. We then close the spurious gap by training new classification heads on lowly ranked (i.e. without common spurious cues) images, resulting in improved effective robustness to distribution shifts (ObjectNet, ImageNet-R, ImageNet-Sketch). We also propose a second metric to assess feature reliability, finding that spurious features are generally less reliable than non-spurious (core) ones, though again, spurious features can be more reliable for certain classes. To enable our analysis, we annotated $5,000$ feature-class dependencies over {\it all} of ImageNet as core or spurious using minimal human supervision. Finally, we show the feature discovery and spuriosity ranking framework can be extended to other datasets like CelebA and WaterBirds in a lightweight fashion with only linear layer training, leading to discovering a previously unknown racial bias in the Celeb-A hair classification.
翻译:我们根据虚伪的提示力,在课堂上展示了一个排名图象的框架。我们通过测量最高和最低级别图象的准确性差距(我们称之为虚假的缺口 ), 评估了89美元多元图像网模型的虚假依赖性,发现即使是最好的模型在图像上表现不佳,而虚假存在的存在也微弱。然而,假的提示效应在各年级之间差异更大得多,强调虚假相关性问题的关键性、经常被忽视的阶级依赖性。虽然我们观察到的多数虚假特征正在澄清(即提高当前测试-时间的准确性,这是通常预期的 ),但我们令人惊讶地发现了许多令人困惑的虚假特征,模型在缺少时表现得更好。我们随后通过在低级别(即没有常见的虚假提示)图像中培训新的分类头来弥补这种虚假的缺口,从而提高了分布变化的有效性(ObjotNet,图像网-R,图像网-Ket-Sketchch ) 。我们建议用第二个指标来评估特征的可靠性,发现虚假特征比非纯性(核心) 直线性(核心) 性) 性特征分析, 最终将Sellialal-dealal-dealalalalal exal exal ex extime extitudududududududududududustrt ex ex ex ex ex ex ex ex ex ex ex