Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. Although prompt tuning has been shown to match the performance of full model tuning when training data is sufficient, it tends to struggle in few-shot learning settings. In this paper, we present Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks. On downstream tasks, the pre-trained prompts are selectively activated and combined, leading to strong compositional generalization to unseen tasks. To bridge the gap between pre-training and fine-tuning, we formulate upstream and downstream tasks into a unified machine reading comprehension task. Extensive experiments under two learning paradigms, i.e., gradient descent and black-box tuning, show that MP2 significantly outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot settings. In addition, we demonstrate that MP2 can achieve surprisingly fast and strong adaptation to downstream tasks by merely learning 8 parameters to combine the pre-trained modular prompts.
翻译:快速调试是使经过培训的语文模式适应下游任务的一种具有参数效率的方法。虽然在培训数据充足时,快速调试已经证明与完全模型调试的性能相匹配,但往往会在几发学习环境中挣扎。在本文件中,我们介绍了多任务预调模块(MP2),以加快对微粒学习的快速调试。MP2是一套对38个中国任务进行预先培训的可燃提示。在下游任务中,预先培训的提示被有选择地激活和组合,导致对看不见任务进行强有力的组合化概括化。为了缩小培训前和微调之间的差距,我们将上游和下游任务发展成一个统一的机器阅读理解任务。在两种学习模式(即梯度下下移和黑盒调)下进行的广泛实验表明,MP2在微粒情况下大大超过快速调、完全模型调试用以及先前的快速培训前方法。此外,我们证明,MP2可以通过仅仅学习8项参数,将经过培训前模块的及时性综合起来,从而对下游任务作出惊人的快速和有力的调整。