Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in knowledge graphs (KGs) and achieved consistent improvements on various knowledge-driven NLP tasks. However, most of these knowledge-enhanced PLMs embed static sub-graphs of KGs ("knowledge context"), regardless of that the knowledge required by PLMs may change dynamically according to specific text ("textual context"). In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text. Our experimental results show that Coke outperforms various baselines on typical knowledge-driven NLP tasks, indicating the effectiveness of utilizing dynamic knowledge context for language understanding. Besides the performance improvements, the dynamically selected knowledge in Coke can describe the semantics of text-related knowledge in a more interpretable form than the conventional PLMs. Our source code and datasets will be available to provide more details for Coke.
翻译:最近,许多工作为增强预训练语言模型 (PLMs) 利用了知识图谱 (KGs) 中的额外异质知识,并在各种知识驱动的 NLP 任务上实现了一致的改进。然而,这些知识增强的 PLMs 大多嵌入 KGs 的静态子图(“知识上下文”),而不考虑 PLMs 根据具体文本(“文本上下文”)动态更改所需的知识。本文提出了一个名为 Coke 的新框架,用于 PLMs 动态选择上下文知识并根据文本上下文嵌入知识上下文,从而可以避免 KGs 中的冗余和模糊知识对输入文本的影响。我们的实验结果表明,Coke 在典型的知识驱动 NLP 任务上优于各种基线,表明利用动态上下文知识进行语言理解的有效性。除了性能提升外,Coke 中的动态选择的知识可以以比传统 PLMs 更可解释的形式描述与文本相关的知识语义。我们的源代码和数据集将提供更多关于 Coke 的详细信息。