Inferring the probability distribution of sentences or word sequences is a key process in natural language processing. While word-level language models (LMs) have been widely adopted for computing the joint probabilities of word sequences, they have difficulty in capturing a context long enough for sentence probability estimation (SPE). To overcome this, recent studies introduced training methods using sentence-level noise-contrastive estimation (NCE) with recurrent neural networks (RNNs). In this work, we attempt to extend it for contextual SPE, which aims to estimate a conditional sentence probability given a previous text. The proposed NCE samples negative sentences independently of a previous text so that the trained model gives higher probabilities to the sentences that are more consistent with \textcolor{blue}{the} context. We apply our method to a simple word-level RNN LM to focus on the effect of the sentence-level NCE training rather than on the network architecture. The quality of estimation was evaluated against multiple-choice cloze-style questions including both human and automatically generated questions. The experimental results show that the proposed method improved the SPE quality for the word-level RNN LM.
翻译:推断句或字序列的概率分布是自然语言处理过程中的一个关键过程。虽然在计算单词序列的共同概率时广泛采用了单词级语言模型(LMs),但是它们很难捕捉到足够长的上下文来进行判决概率估计(SPE)。为了克服这一点,最近的研究采用了使用判决级噪声调估计(NCE)和经常性神经网络(RNNNs)的培训方法。在这项工作中,我们试图将这一方法扩大到背景的SPE,目的是根据先前的文本来估计一个有条件的句概率。拟议的 NCE 模拟否定的句子,独立于以前的文本,这样,经过培训的模型能给更符合\textcolor{blue ⁇ the}背景的句子带来更高的概率。我们运用了简单的单词级 RNNNLM 方法,侧重于判决级NCE培训的影响,而不是网络结构。我们试图根据多重选择式的Cloze类问题,包括人的问题和自动产生的问题来评估估算估算质量。实验结果显示,拟议的方法改进了LNNMLM等级的SPE质量。