Retrieval-based language models are increasingly employed in question-answering tasks. These models search in a corpus of documents for relevant information instead of having all factual knowledge stored in its parameters, thereby enhancing efficiency, transparency, and adaptability. We develop the first Norwegian retrieval-based model by adapting the REALM framework and evaluating it on various tasks. After training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks. Results show that retrieval augmented language modeling improves the reader's performance on extractive question-answering, suggesting that this type of training improves language models' general ability to use context and that this does not happen at the expense of other abilities such as part-of-speech tagging, dependency parsing, named entity recognition, and lemmatization. Code, trained models, and data are made publicly available.
翻译:检索型语言模型在问答任务中越来越被使用。这些模型在文档语料库中搜索相关信息,而不是将所有事实知识存储在其参数中,从而提高了效率、透明度和适应性。我们通过调整REALM框架开发了第一个挪威语检索型模型,并在各种任务上进行了评估。训练完成后,我们将语言模型(称为阅读器)与检索器组件分离,并表明这可以在各种下游任务上进行微调。结果表明,检索增强型语言建模改善了读者在提取式问答方面的表现,表明这种类型的训练提高了语言模型使用上下文的通用能力,但并不以牺牲其他能力(如词性标注、依赖解析、命名实体识别和词形变化)为代价。代码、训练模型和数据已经公开提供。