Pretrained language models are generally acknowledged to be able to encode syntax [Tenney et al., 2019, Jawahar et al., 2019, Hewitt and Manning, 2019]. In this article, we propose UPOA, an Unsupervised constituent Parsing model that calculates an Out Association score solely based on the self-attention weight matrix learned in a pretrained language model as the syntactic distance for span segmentation. We further propose an enhanced version, UPIO, which exploits both inside association and outside association scores for estimating the likelihood of a span. Experiments with UPOA and UPIO disclose that the linear projection matrices for the query and key in the self-attention mechanism play an important role in parsing. We therefore extend the unsupervised models to few-shot parsing models (FPOA, FPIO) that use a few annotated trees to learn better linear projection matrices for parsing. Experiments on the Penn Treebank demonstrate that our unsupervised parsing model UPIO achieves results comparable to the state of the art on short sentences (length <= 10). Our few-shot parsing model FPIO trained with only 20 annotated trees outperforms a previous few-shot parsing method trained with 50 annotated trees. Experiments on cross-lingual parsing show that both unsupervised and few-shot parsing methods are better than previous methods on most languages of SPMRL [Seddah et al., 2013].
翻译:普遍承认,受过训练的语言模型能够对语法进行编码[Tenney等人,2019年,Jawahar等人,2019年,Hawahar等人,2019年,Hewitt和Manning,2019年]。在本篇文章中,我们提出UPOA,这是一个无人监督的组成剖析模型,计算退出协会得分的唯一依据是在预先培训的语言模型中学习的自我注意权重矩阵,作为用于跨段隔段的合成距离。我们进一步提议一个强化版本,UPIO,它利用内部和外部关联得分来估计跨段的可能性。与UPOA和UPIO进行的实验表明,自我注意机制中的查询和关键的线性投影矩阵在解析中起着重要作用。因此,我们将未经监督的模型推广到几张分分的模型(FPOA,FPIO),它使用一些附加说明的树来学习更好的线性预测矩阵。在Penn 树库的实验表明,我们未经监督的模型在前几张UDIO中取得了更好的结果,仅与经过训练的20种直径直判的直径直判的直径直径直径,在前的S-O的直判的直径直径直判中展示的直判方法上展示了一个直径直径10。