Explicit decomposition modeling, which involves breaking down complex tasks into more straightforward and often more interpretable sub-tasks, has long been a central theme in developing robust and interpretable NLU systems. However, despite the many datasets and resources built as part of this effort, the majority have small-scale annotations and limited scope, which is insufficient to solve general decomposition tasks. In this paper, we look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts, particularly large-scale parallel news. We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible. For example, on semantic parsing, our model, DecompT5, improves 20% to 30% on two datasets, Overnight and TORQUE, over the baseline language model. We further use DecompT5 to build a novel decomposition-based QA system named DecompEntail, improving over state-of-the-art models, including GPT-3, on both HotpotQA and StrategyQA by 8% and 4%, respectively.
翻译:显性分解模型涉及将复杂任务分为更简单、往往更易解释的子任务,长期以来一直是发展稳健和可解释的NLU系统的一个中心主题,然而,尽管作为这一努力的一部分建立了许多数据集和资源,但大多数都具有小规模说明和范围有限,不足以解决一般分解任务。在本文件中,我们利用从可比较文本,特别是大规模平行新闻的远程监督,对基于分解的变异器进行大规模中级培训。我们表明,通过这种中期培训,为多种任务开发强有力的基于分解的模型更为可行。例如,在语义分类、我们的模型、DecomPT5方面,在两个数据集(即夜间和TORQUE)上,比基线语言模型提高20%至30%。我们进一步使用DecomT5来建立一个新型的基于分解变的QA系统,名为Decompentail,在热波特QA和战略上,分别提高8%和4%的州级模型,包括GPT-3。