One often wants to take an existing, trained NLP model and use it on data from a new domain. While fine-tuning or few-shot learning can be used to adapt the base model, there is no one simple recipe to getting these working; moreover, one may not have access to the original model weights if it is deployed as a black box. To this end, we study how to improve a black box model's performance on a new domain given examples from the new domain by leveraging explanations of the model's behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, and then uses a simple model to calibrate or rerank the model's predictions based on the features. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains. The experimental results across all the domain pairs show that explanations are useful for calibrating these models. We show that the calibration features transfer to some extent between tasks and shed light on how to effectively use them.
翻译:人们往往希望采用现有、经过培训的NLP模型,并将其用于新域的数据。 虽然可以使用微调或微小的学习来调整基数模型, 但没有一种简单的方法来使这些模型发挥作用; 此外, 如果以黑匣子的形式部署, 一个人可能无法获得原始模型加权数。 为此, 我们研究如何利用新域对模型行为的解释来改进黑盒模型在新域给定实例的新域上的性能。 我们的方法首先提取一套将有关任务的人的直觉与黑盒判读技术生成的模型属性相结合的特征, 然后使用一个简单模型来校准或重新排列基于这些特征的模型预测数。 我们用我们的方法实验了两种任务, 即采掘问题回答和自然语言推论, 包括几个域的调整。 所有域对的实验结果显示, 解释对于校准这些模型是有用的。 我们显示, 校准特征在任务和如何有效使用这些模型的亮度之间, 在某种程度上转移了任务和亮度。