Pre-training methods with contrastive learning objectives have shown remarkable success in dialog understanding tasks. However, current contrastive learning solely considers the self-augmented dialog samples as positive samples and treats all other dialog samples as negative ones, which enforces dissimilar representations even for dialogs that are semantically related. In this paper, we propose SPACE-2, a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora via semi-supervised contrastive pre-training. Concretely, we first define a general semantic tree structure (STS) to unify the inconsistent annotation schema across different dialog datasets, so that the rich structural information stored in all labeled data can be exploited. Then we propose a novel multi-view score function to increase the relevance of all possible dialogs that share similar STSs and only push away other completely different dialogs during supervised contrastive pre-training. To fully exploit unlabeled dialogs, a basic self-supervised contrastive loss is also added to refine the learned representations. Experiments show that our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks. For reproducibility, we release the code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/space-2.
翻译:具有对比性学习目标的培训前方法在对话理解任务中表现出了显著的成功。然而,目前的对比性学习仅仅将自我强化的对话框样本视为正样,并将所有其他对话框样本视为负样,这些样本强制进行不同表达,甚至对于与语义相关的对话也是如此。在本文中,我们提议Space-2,一个树结构化的预培训对话模式,从有限的标签式对话中学习对话演示,通过半监督的对比性培训前阶段大规模无标签对话团团。具体地说,我们首先定义一个通用的语义树结构(STS),以统一不同对话框数据集之间不一致的注解模式,这样所有标签数据中储存的丰富的结构信息都可以被利用。然后我们提出一个新的多视角评分功能,以提高所有共享类似 TSS 的可能的对话的相关性,而在监督的对比性培训前阶段,只能将其他完全不同的对话推开。为了充分利用未标签式的大众对话,一个基本的自我监督性对比性损失也添加到完善学习性的表达式结构图中, 实验显示我们的方法可以实现新的数据/预定式数据格式。