Can we create a question answering (QA) dataset that, by construction, requires proper multi-hop reasoning? This goal has been surprisingly elusive. We introduce a bottom-up approach that systematically selects composable pairs of single-hop questions that are connected, i.e., where one reasoning step requires information from the other. This bottom-up approach allows greater control over the properties of the resulting $k$-hop questions. We add stringent filters and other mechanisms targeting connected reasoning, including minimizing many forms of train-test leakage, improved distractor contexts, and contrasting unanswerable questions at the sub-question level. We use this process to construct MuSiQue-Ans, a new multihop QA dataset with 25K 2-4 hop questions, built using seed questions from 5 existing single-hop datasets. Our experiments demonstrate that MuSiQue-Ans is challenging for state-of-the-art QA models significantly harder than existing datasets (3x human-machine gap in a comparable setting), and substantially less cheatable (e.g., a single-hop model is worse by 30 F1 pts). We also build a more challenging dataset, MuSiQue-Full, consisting of answerable and unanswerable contrast question pairs, where model performance drops further by 14 F1 pts.
翻译:我们能否创建一个问题解答(QA)数据集? 通过构建这样的解答(QA)数据集, 需要适当的多点推理? 这个目标令人惊讶地难以实现。 我们引入了一种自下而上的方法, 系统地选择相配的单点问题, 即一个推理步骤需要从另一点获得信息。 这种自下而上的方法可以对由此产生的$k$- hop问题的属性进行更大的控制。 我们添加了严格的过滤器和其他机制, 以相关推理为目标, 包括尽量减少多种形式的火车测试渗漏、 改进分散开关环境, 和在子问题级别上对比无法回答的问题。 我们使用这个程序来构建 MusiQA 数据集, 一个新的多点QA数据集, 包含 25K 2-4 跳问题 。 我们的实验表明, MusiQue- Ans 模型比现有的最新QA模型( 3x 人类机器差距) 还要大得多,, 并且比现有的数据集要小得多( 例如, 单点模型比的F1 样更难 ) 。