Pre-trained contextual representations have led to dramatic performance improvements on a range of downstream tasks. This has motivated researchers to quantify and understand the linguistic information encoded in them. In general, this is done by probing, which consists of training a supervised model to predict a linguistic property from said representations. Unfortunately, this definition of probing has been subject to extensive criticism, and can lead to paradoxical or counter-intuitive results. In this work, we present a novel framework for probing where the goal is to evaluate the inductive bias of representations for a particular task, and provide a practical avenue to do this using Bayesian inference. We apply our framework to a series of token-, arc-, and sentence-level tasks. Our results suggest that our framework solves problems of previous approaches and that fastText can offer a better inductive bias than BERT in certain situations.
翻译:未经培训的背景介绍导致一系列下游任务的业绩显著改善,这促使研究人员量化和理解其中所编码的语言信息。一般而言,这是通过调查实现的,调查包括培训一种监督模型,以预测上述表述中的语言财产。不幸的是,这种调查的定义受到广泛批评,可能导致自相矛盾或反直觉的结果。在这项工作中,我们提出了一个新的调查框架,目的是评估某一特定任务中代表的诱导偏差,并提供一个实用的途径,利用巴耶斯语的推理进行这项工作。我们将我们的框架应用于一系列象征性、弧和判决层面的任务。我们的结果表明,我们的框架解决了以前方法的问题,快速技术在某些情况下可以提供比BERT更好的诱导偏差。