Interleaving is an online evaluation approach for information retrieval systems that compares the effectiveness of ranking functions in interpreting the users' implicit feedback. Previous work such as Hofmann et al (2011) has evaluated the most promising interleaved methods at the time, on uniform distributions of queries. In the real world, ordinarily, there is an unbalanced distribution of repeated queries that follows a long-tailed users' search demand curve. The more a query is executed, by different users (or in different sessions), the higher the probability of collecting implicit feedback (interactions/clicks) on the related search results. This paper first aims to replicate the Team Draft Interleaving accuracy evaluation on uniform query distributions and then focuses on assessing how this method generalizes to long-tailed real-world scenarios. The reproducibility work raised interesting considerations on how the winning ranking function for each query should impact the overall winner for the entire evaluation. Based on what was observed, we propose that not all the queries should contribute to the final decision in equal proportion. As a result of these insights, we designed two variations of the $\Delta_{AB}$ score winner estimator that assign to each query a credit based on statistical hypothesis testing. To replicate, reproduce and extend the original work, we have developed from scratch a system that simulates a search engine and users' interactions from datasets from the industry. Our experiments confirm our intuition and show that our methods are promising in terms of accuracy, sensitivity, and robustness to noise.
翻译:Stat-weight: 使用统计假设检验改进交替方法结果评估估计器
Translated Abstract:
交替评价是信息检索系统的在线评估方法,它比较排名函数在解释用户隐性反馈方面的效果。 Hofmann等人 (2011)等之前的研究已经评估了最有前途的交替方法,但仅仅是在均匀分布的查询上。在现实世界中,通常会有一个遵循长尾用户搜索需求曲线的查询无序分布。当查询被不同的用户(或在不同的会话中)重复执行时,收集隐式反馈(交互/点击)的概率越高。本文的第一目标是在统一查询分布上复制Team Draft Interleaving精度评估,然后重点评估该方法在长尾实际场景下的泛化能力。再次重现研究结果的过程中,我们对每个查询的胜出排名函数对整个评估的胜出排名函数的影响产生了有趣的考虑。基于我们的观察,我们建议不应该所有查询都以相等的比例对最终决策做出贡献。作为我们的洞察力的一部分,我们设计了两种变体的$\Delta_{AB}$得分获胜者估计器,它们基于统计假设检验为每个查询分配信用。为了复制、再现和扩展原始研究,我们从行业数据集中自己编写了一个模拟搜索引擎和用户交互的系统。我们的实验验证了我们的直觉,并表明我们的方法在准确性、灵敏度和对噪音的稳健性方面是有前途的。