Pretrained using large amount of data, autoregressive language models are able to generate high quality sequences. However, these models do not perform well under hard lexical constraints as they lack fine control of content generation process. Progressive insertion-based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input tokens as constraint. These transformers however may fail to support hard lexical constraints as their generation process is more likely to terminate prematurely. The paper analyses such early termination problems and proposes the Entity-constrained insertion transformer (ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation efficiency. We introduce a new training strategy that considers predefined hard lexical constraints (e.g., entities to be included in the generated sequence). Our experiments show that ENCONTER outperforms other baseline models in several performance metrics rendering it more suitable in practical applications. Our code is available at https://github.com/LARC-CMU-SMU/Enconter
翻译:使用大量数据的预先训练,自动递减语言模型能够产生高质量的序列。 但是,这些模型在硬词汇限制下效果不佳,因为它们缺乏对内容生成过程的精密控制。 渐进式插入式变压器可以克服上述限制,并且以某些输入符号作为约束,能够有效地平行生成序列。 然而,这些变压器可能无法支持硬词汇限制,因为它们的生成过程更有可能提前终止。 文件分析早期终止问题,并提议采用实体限制的插入变压器(ENCONTER),这是一个新的插入变压器,可以解决上述陷阱,同时又不损害生成效率。 我们引入了新的培训战略,考虑预先定义的硬词汇限制(例如,实体将包括在生成序列中)。 我们的实验显示, ENCONTER(ENCONTER)在几项性能衡量标准中优于其他基线模型,使其更适合实际应用。我们的代码可在https://github.com/LARC-CMU-SMU/Enconter上查阅。