With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, various fairness metrics have been proposed but they do not account for the user utility or do not measure it adequately. To address this problem, we propose a new metric called Fairness-Aware IR (FAIR). By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness-aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to existing metrics and demonstrated the effectiveness of our FAIR-based algorithm. We believe our work opens up a new direction of pursuing a computationally feasible metric for evaluating and implementing the fairness-aware IR systems.
翻译:由于新出现的需要为搜索和建议系统创造公平意识的解决方案,评估这类解决方案是一项艰巨的挑战。虽然许多传统的信息检索(IR)衡量标准能够捕捉用户效用的相关性、多样性和新颖性,但它们不适合从负责任的信息曝光的角度来推断列报的结果是否公平。另一方面,提出了各种公平度指标,但没有考虑到用户效用,也没有适当计量。为了解决这个问题,我们提议了一个新的衡量标准,称为公平-Aware IR(FAIR)。通过将标准IR衡量标准和公平性措施统一为综合衡量标准,这一衡量标准为评估公平意识评级结果提供了新的视角。根据这一衡量标准,我们制定了有效的排序算法,共同优化用户效用和公平性。实验结果表明,我们的公平性衡量标准可以以良好的用户效用和公平的信息曝光来突出结果。我们展示了财务IR与现有衡量标准的关系,并展示了我们基于公平性IR的算法的有效性。我们相信我们的工作开辟了一种新的方向,即为评估和实施公平性的IR系统而追求一种计算可行的衡量标准。