Industrial ranking systems, such as advertising systems, rank items by aggregating multiple objectives into one final objective to satisfy user demand and commercial intent. Cascade architecture, composed of retrieval, pre-ranking, and ranking stages, is usually adopted to reduce the computational cost. Each stage may employ various models for different objectives and calculate the final objective by aggregating these models' outputs. The multi-stage ranking strategy causes a new problem - the ranked lists of the ranking stage and previous stages may be inconsistent. For example, items that should be ranked at the top of the ranking stage may be ranked at the bottom of previous stages. In this paper, we focus on the ranking consistency between the pre-ranking and ranking stages. Specifically, we formally define the problem of ranking consistency and propose the Ranking Consistency Score (RCS) metric for evaluation. We demonstrate that ranking consistency has a direct impact on online performance. Compared with the traditional evaluation manner that mainly focuses on the individual ranking quality of every objective, RCS considers the ranking consistency of the fused final objective, which is more proper for evaluation. Finally, to improve the ranking consistency, we propose several methods from the perspective of sample selection and learning algorithms. Experimental results on industrial datasets validate the efficacy of the proposed metrics and methods. The proposed consistency methods have been deployed on the display advertising system of Alibaba, obtaining a 6.7% improvement on CTR (Click-Through Rate) and a 5.5% increase on RPM (Revenue Per Mille).
翻译:产业排名系统,如广告系统,通过将多个目标合并成一个最终目标,以满足用户需求和商业意图等产业排名项目,将多个目标合并成一个最终目标,从而满足用户需求和商业意图。通常采用由检索、排名前和排名阶段组成的连锁结构来降低计算成本。每个阶段都可以为不同目标采用不同的模型,并通过汇总这些模型的产出来计算最终目标。多阶段排名战略造成了一个新问题――排名阶段和以前阶段的排名清单可能不一致。例如,排名排名前和排名阶段之间的排名顺序排列可能排在前几个阶段的底端。在本文件中,我们侧重于排名前和排名阶段之间的排序一致性。具体地说,我们正式界定排名一致性问题,并提出排名一致评分标准,供评价使用。我们表明,排名的一致性对在线业绩有直接影响。与主要侧重于每个目标的排位质量的传统评价方式相比,RCS认为统一的最后目标的排位一致性可能排在前几个阶段的底端。最后目标的排在下。最后评分,对于改进排名的一致性,最后,我们从抽样选择和排序前等级等级等级等级等级分级阶段和排名阶段的等级分级阶段之间,我们正式提出一些方法,我们从抽样选择的排序问题的角度确定一致性问题,并提议的排名顺序排序顺序排序顺序排序顺序排序。