Words of estimative probability (WEP) are expressions of a statement's plausibility (probably, maybe, likely, doubt, likely, unlikely, impossible...). Multiple surveys demonstrate the agreement of human evaluators when assigning numerical probability levels to WEP. For example, highly likely corresponds to a median chance of 0.90+-0.08 in Fagen-Ulmschneider (2015)'s survey. In this work, we measure the ability of neural language processing models to capture the consensual probability level associated to each WEP. Firstly, we use the UNLI dataset (Chen et al., 2020) which associates premises and hypotheses with their perceived joint probability p, to construct prompts, e.g. "[PREMISE]. [WEP], [HYPOTHESIS]." and assess whether language models can predict whether the WEP consensual probability level is close to p. Secondly, we construct a dataset of WEP-based probabilistic reasoning, to test whether language models can reason with WEP compositions. When prompted "[EVENTA] is likely. [EVENTB] is impossible.", a causal language model should not express that [EVENTA&B] is likely. We show that both tasks are unsolved by off-the-shelf English language models, but that fine-tuning leads to transferable improvement.
翻译:估计概率( WEP) 的字数表示声明的可信度( 可能、 可能、 可能、 怀疑、 可能、 可能、 不可能. ) 。 多项调查显示, 在给 WEP 分配数值概率水平时, 人类评价员同意 。 例如, 在 Fagen- Ulmschneider (2015) 的调查中, 极有可能对应0. 90+- 0.08 的中位概率值 。 在这项工作中, 我们测量神经语言处理模型的能力, 以捕捉与每个 WEP 相关的一致概率水平。 首先, 我们使用UNLI 数据集( Chen et al., 2020) 来将房地和假设与其感知的共同概率( ) 相联的数据集( p) 。 例如, “ [ PREMISE]. [WEP, [HYPSIS], [HYPSUPSB] 和 BYLULA 都不可能 改进语言模式。