By explaining how humans would solve a given task, human rationales can provide strong learning signal for neural language models (LMs). Explanation regularization (ER) aims to improve LM generalization by pushing the LM's machine rationales (Which input tokens did the LM focus on?) to align with human rationales (Which input tokens would humans focus on?). Though prior works primarily study ER via in-distribution (ID) evaluation, out-of-distribution (OOD) generalization is often more critical in real-world scenarios, yet ER's effect on OOD generalization has been underexplored. In this paper, we introduce ER-Test, a framework for evaluating ER models' OOD generalization along three dimensions: unseen dataset tests, contrast set tests, and functional tests. Using ER-Test, we extensively analyze how ER models' OOD generalization varies with different ER design choices. Across two tasks and six datasets, ER-Test shows that ER has little impact on ID performance but can yield large OOD performance gains. Also, we find that ER can improve OOD performance even with limited rationale supervision. ER-Test's results help demonstrate ER's utility and establish best practices for using ER effectively.
翻译:通过解释人类如何解决某一任务,人类的原理可以为神经语言模型(LMS)提供强有力的学习信号。解释规范(ER)的目的是通过推动LM的机器原理(LM关注的输入符号? )与人的理由(人类关注的输入符号?? )保持一致,从而改进LM的机械原理(LM的输入符号? ),从而与人的理由(人类关注的?? )相一致。虽然先前的工作主要是通过分布(ID)评价研究ER,但分配外的概括化在现实世界情景中往往更为关键,然而ER对OOD一般化的影响却没有得到充分的探讨。在本文中,我们引入ER-Test,这是评估ER模型对 OODD一般化的评估框架,从三个方面来说是:隐秘的数据集测试、对比设置测试以及功能测试。我们利用ER-Test,广泛分析ER模型的OODG的概括化与不同的设计选择有何不同。在两个任务和六个数据集中,ER-Test显示ER对ID的绩效影响很小,但能够产生巨大的OD绩效收益。此外,我们发现,利用ODER-ER的实用性实践可以有效地证明OD-ERsurgals。</s>