Retrieval-Augmented Language Modeling (RALM) methods, that condition a language model (LM) on relevant documents from a grounding corpus during generation, have been shown to significantly improve language modeling while also providing a natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper proposes an under-explored alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input. We show that in-context RALM which uses off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that in-context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access. To that end, we make our code publicly available.
翻译:现有语言模型方法侧重于修改LM结构,以便利纳入外部信息,使部署工作更加复杂化。本文提出了一个探索不足的替代方法,即我们把LM结构放在Context RALM上:将LM结构保持不变,将地面文件放在输入中。我们显示,使用现成一般用途检索器的文本RALM在模型大小和多种组合中带来了令人惊讶的LM大增益。我们还表明,文件检索和排序机制可以专门用于RALM环境,以进一步提升性能。我们的结论是,文中RALM有很大潜力增加LM地基的普及,特别是在必须不经修改或甚至通过API访问而使用预先训练的LM的环境下。为此,我们公开提供我们的代码。