Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) systems. In this paper, we showcase the advantages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks. To this end, we utilize light-weight adapters that can be easily integrated with PLMs and serve as a repository for facts learned from different KBs. To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) -- a probe designed specifically for TOD models. Experiments on KPRS and the response generation task show improvements of knowledge injection with adapters over strong baselines.
翻译:培训前语言模型(PLM)在全非学习前应用中提高了最新水平,但缺乏在培训前数据中自然不会发生的特定领域知识。以前的研究为下游国家学习计划的不同任务增加了具有象征意义的知识的PLM。然而,这些研究中使用的知识基础(KBs)通常是大规模和静态的,与小型、特定领域和可调整的知识基础不同,后者在现实世界的任务导向对话(TOD)系统中占有突出地位。在本文中,我们展示了在对TOD任务进行微调之前,注射特定领域知识的优势。为此,我们利用轻量适应器,很容易与PLMs结合,并充当从不同KBs学到的事实的存放处。为了衡量拟议的知识注入方法的效力,我们引入了使用反应选择的知识检验(KPRS) -- -- 一种专门为TOD模型设计的探测器。对KPRS和反应生成任务实验显示与适应者一起在强基线上进行知识注入的改进。