We investigate whether three types of post hoc model explanations--feature attribution, concept activation, and training point ranking--are effective for detecting a model's reliance on spurious signals in the training data. Specifically, we consider the scenario where the spurious signal to be detected is unknown, at test-time, to the user of the explanation method. We design an empirical methodology that uses semi-synthetic datasets along with pre-specified spurious artifacts to obtain models that verifiably rely on these spurious training signals. We then provide a suite of metrics that assess an explanation method's reliability for spurious signal detection under various conditions. We find that the post hoc explanation methods tested are ineffective when the spurious artifact is unknown at test-time especially for non-visible artifacts like a background blur. Further, we find that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals even when the model being explained does not rely on spurious artifacts. This finding casts doubt on the utility of these approaches, in the hands of a practitioner, for detecting a model's reliance on spurious signals.
翻译:我们调查三种类型的后特设模型解释-特性归属、概念激活和培训点排名是否有效,以发现模型依赖培训数据中的虚假信号。 具体地说,我们考虑在测试时,解释方法的使用者不知道要检测到的虚假信号的情景。 我们设计了一种经验性方法,使用半合成数据集以及预先确定的虚假文物,以获得可核实地依赖这些虚假培训信号的模型。 然后,我们提供了一套衡量尺度,评估在各种条件下对虚假信号探测的解释方法的可靠性。 我们发现,在测试时,特别是对于背景模糊的不可见的文物,所测试的后特设解释方法是无效的。 此外,我们发现特性归属方法很容易错误地表明,即使解释模型并不依赖虚假的手工艺,对虚假信号的依赖性信号的依赖性。 这使人怀疑这些方法在执业者手中是否有用,用以检测模型对虚假信号的依赖性。