We propose reconstruction probing, a new analysis method for contextualized representations based on reconstruction probabilities in masked language models (MLMs). This method relies on comparing the reconstruction probabilities of tokens in a given sequence when conditioned on the representation of a single token that has been fully contextualized and when conditioned on only the decontextualized lexical prior of the model. This comparison can be understood as quantifying the contribution of contextualization towards reconstruction -- the difference in the reconstruction probabilities can only be attributed to the representational change of the single token induced by contextualization. We apply this analysis to three MLMs and find that contextualization boosts reconstructability of tokens that are close to the token being reconstructed in terms of linear and syntactic distance. Furthermore, we extend our analysis to finer-grained decomposition of contextualized representations, and we find that these boosts are largely attributable to static and positional embeddings at the input layer.
翻译:我们建议进行重建预测,这是基于隐蔽语言模型中重建概率的根据重建概率进行背景化表述的新分析方法。这种方法依赖于比较某一序列中象征物的重建概率,条件是只代表一个已经完全符合背景的单一象征物,且仅以该模型之前的解析法为条件。这一比较可以理解为量化背景化对重建的贡献 -- -- 重建概率的差异只能归因于因背景化引起的单一象征物的代表性变化。我们将这一分析应用到3个MLMs,并发现背景化可以促进正在重建的线性和合成距离接近象征物的象征物的可重建性。此外,我们将我们的分析扩大到背景化代表物的细化分解,我们发现这些增强力在很大程度上归因于输入层的静态和定位嵌入。