Recently, pretrained language models (PLMs) have had exceptional success in language generation. To leverage the rich knowledge encoded by PLMs, a simple yet powerful paradigm is to use prompts in the form of either discrete tokens or continuous embeddings. In existing studies, these prompting methods are typically independent of the inputs, lacking sufficient consideration of input semantics. To address this issue, we propose a novel continuous prompting approach, called context-tuning, to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text to elicit useful knowledge from PLMs for generation. We refer to such prompts as contextualized prompts. Secondly, we use continuous inverse prompting to improve the process of natural language generation by modeling an inverse generation process from output to input, making the generated text more relevant to the inputs. Furthermore, we utilize a lightweight context-tuning method that fine-tunes only 0.12% of the parameters while maintaining good performance. Our code is publicly available at https://github.com/RUCAIBox/Context-Tuning.
翻译:最近,经过培训的语言模型(PLMs)在语言生成方面取得了非凡的成功。为了利用PLM所编码的丰富知识,一个简单而有力的范例是使用离散符号或连续嵌入的提示。在现有研究中,这些提示方法通常独立于投入,没有充分考虑到投入的语义。为了解决这一问题,我们提议了一种新型的连续不断提示方法,称为背景调整,以微调天然语言生成的PLMs。首先,根据输入文本生成提示,以便从PLMs中获取有用的知识。我们提到背景化提示等提示。第二,我们不断使用反向提示,通过模拟从输出到输入的反向生成过程来改进自然语言生成过程,使生成的文字与投入更加相关。此外,我们使用一种轻量的背景调整方法,在保持良好性能的同时只微调0.12%的参数。我们的代码可在https://github.com/RUCAIBox/Context-Tuning上公开查阅。