Data augmentations (DA) are the cores to achieving robust sequence-to-sequence learning on various natural language processing (NLP) tasks. However, most of the DA approaches force the decoder to make predictions conditioned on the perturbed input representation, underutilizing supervised information provided by perturbed input. In this work, we propose a framework-level robust sequence-to-sequence learning approach, named BLISS, via self-supervised input representation, which has the great potential to complement the data-level augmentation approaches. The key idea is to supervise the sequence-to-sequence framework with both the \textit{supervised} ("input$\rightarrow$output") and \textit{self-supervised} ("perturbed input$\rightarrow$input") information. We conduct comprehensive experiments to validate the effectiveness of BLISS on various tasks, including machine translation, grammatical error correction, and text summarization. The results show that BLISS outperforms significantly the vanilla Transformer and consistently works well across tasks than the other five contrastive baselines. Extensive analyses reveal that BLISS learns robust representations and rich linguistic knowledge, confirming our claim. Source code will be released upon publication.
翻译:数据增强( DA) 是实现关于各种自然语言处理( NLP) 任务的稳健顺序到顺序学习的核心。 但是, 大多数 DA 方法都迫使解码器根据受扰动输入的表述方式作出预测, 未充分利用受扰动输入提供的受监督的信息。 在这项工作中, 我们提出一个框架级强势顺序到顺序学习方式, 名为 BLISS, 以自我监督的投入表述方式验证BLISS在各种任务上的有效性, 包括机器翻译、 语法错误更正和文本总结。 关键的想法是监督顺序到顺序框架, 以\ trextit{ supervision} ( “ input$\ drightrow$Uput” ) 和\ textitroutit{ suffer- supervision} ( “ perturmplate put $\rightarrown comput”) 信息。 我们进行全面实验, 以验证 BLISSIS 在各种任务上的有效性, 包括机器翻译、 地校正校正错误校正校正校准, 和不断 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校正 校对 校对 校对 校对 校对 校对 校对 校对 校对 校对 校对 校对 。