Pre-trained Language Models (PLMs) which are trained on large text corpus through the self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Incorporating knowledge into PLMs has been tried to tackle these issues. In this paper, we present a comprehensive review of Knowledge-Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight the focus of these two kinds of tasks. For NLU, we take several types of knowledge into account and divide them into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.
翻译:通过自我监督的学习方法进行大量文字材料培训的经过预先培训的语言模型(PLM)在大量文字材料上取得了在自然语言处理(NLPM)中各种任务方面的有希望的成绩。然而,虽然具有巨大参数的PLM能够有效地掌握从大规模培训文本中学到的丰富知识,并在微调阶段使下游任务受益,但是由于缺乏外部知识,它们仍然有一些局限性,例如推理能力差。将知识纳入PLM(PLM)以解决这些问题。在本文件中,我们全面审查了知识强化的经过培训的预先语言模型(KE-PLMS),以明确了解这个繁荣的领域。我们分别为自然语言理解(NLU)和自然语言生成(NLG)引进了适当的分类,以突出这两类任务的重点。关于NLU,我们考虑到几种知识,并将其分为四类:语言知识、文字知识、知识图(KG)和规则知识知识。我们把NLG的KE-PLMS(K-PLM)分类为基于和检索的方法。我们指出KPLMS的未来方向。