Many natural language inference (NLI) datasets contain biases that allow models to perform well by only using a biased subset of the input, without considering the remainder features. For instance, models are able to make a classification decision by only using the hypothesis, without learning the true relationship between it and the premise. These structural biases lead discriminative models to learn unintended superficial features and to generalize poorly out of the training distribution. In this work, we reformulate the NLI task as a generative task, where a model is conditioned on the biased subset of the input and the label and generates the remaining subset of the input. We show that by imposing a uniform prior, we obtain a provably unbiased model. Through synthetic experiments, we find that this approach is highly robust to large amounts of bias. We then demonstrate empirically on two types of natural bias that this approach leads to fully unbiased models in practice. However, we find that generative models are difficult to train and they generally perform worse than discriminative baselines. We highlight the difficulty of the generative modeling task in the context of NLI as a cause for this worse performance. Finally, by fine-tuning the generative model with a discriminative objective, we reduce the performance gap between the generative model and the discriminative baseline, while allowing for a small amount of bias.
翻译:许多自然语言推论(NLI)数据集包含偏见,使得模型只能使用输入的偏差子集,而没有考虑其余的特征,才能很好地运行。例如,模型只能通过使用假设来作出分类决定,而不了解其真实关系和前提。这些结构性偏见导致歧视模式,以学习无意的表面特征,并将培训分布不均匀。在这项工作中,我们重新将NLI任务改写为归正性任务,以输入和标签的偏差子集为条件,并生成输入的剩余子集。我们表明,通过采用统一的先行模式,我们获得一个可察觉的不偏向模式。我们通过合成实验发现,这一方法非常健全,在很大程度上具有偏向性。然后我们从经验上展示了两种类型的自然偏向性模式,这种模式在实践中导致完全不偏向的模式。然而,我们发现,归正模型很难进行训练,而且通常比歧视基线要差得多。我们强调,在NLI背景下的归正性模型任务很难成为造成这一更差的模型。我们通过精细的基因偏向性模型缩小模型,同时,使我们缩小了这种偏差的基因偏向性模型在目标之间缩小了差距,我们缩小了差距。