Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs. The benchmark datasets, evaluation script, and baseline code to reproduce our results are available at https://github.com/joeljang/continual-knowledge-learning.
翻译:大型语言模型(LMS)将世界知识纳入其参数中,因为它们预示着大量的网络知识库(CKL),常常用于执行基于知识的下游任务,例如回答问题、核对事实和公开对话。在现实世界情景中,LMS中储存的世界知识可以随着世界的变化而迅速过时,但通过广泛的实验,我们发现CKL在避免灾难性的遗忘和可靠地获取新知识的同时保存变化中的知识是非边际的。为了推动社区更好地维护不断变化的LMS,我们制定了一个新的持续学习问题,称为CKL知识学习。我们建立了一个新的基准和衡量标准,以量化保留不变化中的世界知识、更新过时的知识以及获取新知识。我们从文献中采用适用的最新方法来创建几个强有力的基线。我们发现CKLLT展示了以前CL系统设置中未涉及的独特挑战,需要扩展参数来可靠地保留和学习知识。我们通过强调知识的关键性原因,我们显示CKLLL是一个具有挑战性和重要性的问题,能够帮助我们更好地进行基础/学习。