This paper proposes a transformer over transformer framework, called Transformer$^2$, to perform neural text segmentation. It consists of two components: bottom-level sentence encoders using pre-trained transformers, and an upper-level transformer-based segmentation model based on the sentence embeddings. The bottom-level component transfers the pre-trained knowledge learned from large external corpora under both single and pair-wise supervised NLP tasks to model the sentence embeddings for the documents. Given the sentence embeddings, the upper-level transformer is trained to recover the segmentation boundaries as well as the topic labels of each sentence. Equipped with a multi-task loss and the pre-trained knowledge, Transformer$^2$ can better capture the semantic coherence within the same segments. Our experiments show that (1) Transformer$^2$ manages to surpass state-of-the-art text segmentation models in terms of a commonly-used semantic coherence measure; (2) in most cases, both single and pair-wise pre-trained knowledge contribute to the model performance; (3) bottom-level sentence encoders pre-trained on specific languages yield better performance than those pre-trained on specific domains.
翻译:本文建议用变压器框架上的一个变压器,称为变压器$2美元,用于神经文字分解。它由两个部分组成:使用培训前变压器的底部句子编码器,以及基于句内嵌的高级变压器分解模型。底部部分将单一和双向监督的大型外部公司在单一和双向监督的NLP任务下获得的经过培训的知识转换为文档嵌入的句子模型。根据嵌入的句子,高层变压器受过培训,以恢复分解界限和每个句子的主题标签。装备了多任务损失和预先培训的知识,变压器$2美元可以更好地捕捉同一部分内的语义一致性。我们的实验表明:(1)变压器$2美元在常用的语义一致性措施中超过了最先进的文本分解模型;(2)在大多数情况下,单级和配对准的预培训前知识都有助于模型的性能;(3) 低层次的句子在特定语言前的测试前,用于这些特定语言的性能。