Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining

To explain NLP models, importance measures such as attention inform which inputs tokens are important for a prediction are popular. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose an new faithfulness benchmark called Recursive ROAR. This works by recursively masking allegedly important tokens and then retrain the model. The principle is, that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using the area-between-curves, which allows for easy comparison across papers, models, and tasks. To provide a thorough review, we evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.