Learning to rank systems has become an important aspect of our daily life. However, the implicit user feedback that is used to train many learning to rank models is usually noisy and suffered from user bias (i.e., position bias). Thus, obtaining an unbiased model using biased feedback has become an important research field for IR. Existing studies on unbiased learning to rank (ULTR) can be generalized into two families-algorithms that attain unbiasedness with logged data, offline learning, and algorithms that achieve unbiasedness by estimating unbiased parameters with real-time user interactions, namely online learning. While there exist many algorithms from both families, there lacks a unified way to compare and benchmark them. As a result, it can be challenging for researchers to choose the right technique for their problems or for people who are new to the field to learn and understand existing algorithms. To solve this problem, we introduced ULTRA, which is a flexible, extensible, and easily configure ULTR toolbox. Its key features include support for multiple ULTR algorithms with configurable hyperparameters, a variety of built-in click models that can be used separately to simulate clicks, different ranking model architecture and evaluation metrics, and simple learning to rank pipeline creation. In this paper, we discuss the general framework of ULTR, briefly describe the algorithms in ULTRA, detailed the structure, and pipeline of the toolbox. We experimented on all the algorithms supported by ultra and showed that the toolbox performance is reasonable. Our toolbox is an important resource for researchers to conduct experiments on ULTR algorithms with different configurations as well as testing their own algorithms with the supported features.
翻译:学习排名系统已经成为我们日常生活的一个重要方面。 但是,用于培训许多学习排名模型的隐性用户反馈通常很吵,而且受到用户偏差的影响(即位置偏差 ) 。 因此,利用偏差反馈获得一个不偏倚的模式已成为IR的一个重要研究领域。 现有的关于公正学习排名系统(LUCR)的研究可以被广泛分为两个家庭等级,通过登录数据、离线学习和计算法实现公正,通过估算实时用户互动(即在线学习)的公正参数来实现公正。 虽然两个家庭都有许多合理的计算方法,但缺乏统一的方法来比较和衡量它们。 因此,对于研究人员来说,利用偏颇的反馈获得不偏差的模式来选择解决他们的问题的正确技术,或者对新进入实地的人来说,学习和理解现有的算法。 为了解决这个问题,我们引入了“LUCOR”,这是一个灵活、可扩展和易于配置的LETR工具箱。 它的关键特征包括支持多个包含可配置的超直径计的计算方法, 缺乏统一的计算方法, 并且缺乏统一的计算方法。