A notable challenge in Multi-Document Summarization (MDS) is the extremely-long length of the input. In this paper, we present an extract-then-abstract Transformer framework to overcome the problem. Specifically, we leverage pre-trained language models to construct a hierarchical extractor for salient sentence selection across documents and an abstractor for rewriting the selected contents as summaries. However, learning such a framework is challenging since the optimal contents for the abstractor are generally unknown. Previous works typically create pseudo extraction oracle to enable the supervised learning for both the extractor and the abstractor. Nevertheless, we argue that the performance of such methods could be restricted due to the insufficient information for prediction and inconsistent objectives between training and testing. To this end, we propose a loss weighting mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle, and leverage the fine-tuned abstractor to generate summary references as auxiliary signals for learning the extractor. Moreover, we propose a reinforcement learning method that can efficiently apply to the extractor for harmonizing the optimization between training and testing. Experiment results show that our framework substantially outperforms strong baselines with comparable model sizes and achieves the best results on the Multi-News, Multi-XScience, and WikiCatSum corpora.
翻译:多文件摘要化(MDS)的一个显著挑战是投入的篇幅过长。 在本文中,我们提出了一个精选的抽象变异器框架,以克服问题。具体地说,我们利用预先培训的语言模型,为文件的突出刑罚选择建立一个等级抽取器,为将选定内容改写为摘要而建立一个抽象的抽象体,但是,学习这样一个框架具有挑战性,因为对于抽象体的最佳内容一般都不了解。以往的工作通常会创造假提取或奇迹,以便能够对提取者和抽象体进行有监督的学习。然而,我们认为,由于用于预测的信息不足以及培训和测试之间目标不一致,这些方法的性能可能会受到限制。为此,我们提议了一个减重权机制,使模型意识到非在假提取或缩格中的判决的不平等重要性,并利用经过精细调整的抽象体作为学习精选器的辅助信号。此外,我们提议一种强化学习方法,可以有效地适用于提取器,以协调培训和测试之间的优化。实验结果表明,我们的框架大大超越了矩阵的强度基线,具有可比较的多基体规模,并实现了最佳的模型。