The recent MSMARCO passage retrieval collection has allowed researchers to develop highly tuned retrieval systems. One aspect of this data set that makes it distinctive compared to traditional corpora is that most of the topics only have a single answer passage marked relevant. Here we carry out a "what if" sensitivity study, asking whether a set of systems would still have the same relative performance if more passages per topic were deemed to be "relevant", exploring several mechanisms for identifying sets of passages to be so categorized. Our results show that, in general, while run scores can vary markedly if additional plausible passages are presumed to be relevant, the derived system ordering is relatively insensitive to additional relevance, providing support for the methodology that was used at the time the MSMARCO passage collection was created.
翻译:最近的MSMARCO通道检索收集使研究人员能够开发高度调适的检索系统。这一数据集使其与传统公司相比具有独特性的一个方面是,大多数专题只有一个标有相关标记的单一答案。我们在这里进行了“如果”敏感度研究,询问如果每个专题的更多段落被视为“相关”,一套系统是否仍然具有相同的相对性能。我们的结果显示,一般来说,如果认为其他可信的段落具有相关性,则得分会有很大差异,衍生的系统订购相对不敏感于其他关联性,为MSMARCO通道收集时使用的方法提供支持。