Interpreting the performance results of models that attempt to realize user behavior in platforms that employ recommenders is a big challenge that researchers and practitioners continue to face. Although current evaluation tools possess the capacity to provide solid general overview of a system's performance, they still lack consistency and effectiveness in their use as evident in most recent studies on the topic. Current traditional assessment techniques tend to fail to detect variations that could occur on smaller subsets of the data and lack the ability to explain how such variations affect the overall performance. In this article, we focus on the concept of data clustering for evaluation in recommenders and apply a neighborhood assessment method for the datasets of recommender system applications. This new method, named neighborhood-based evaluation, aids in better understanding critical performance variations in more compact subsets of the system to help spot weaknesses where such variations generally go unnoticed with conventional metrics and are typically averaged out. This new modular evaluation layer complements the existing assessment mechanisms and provides the possibility of several applications to the recommender ecosystem such as model evolution tests, fraud/attack detection and a possibility for hosting a hybrid model setup.
翻译:对试图在采用推荐人的平台上实现用户行为的模型的绩效结果进行解释,是研究人员和从业人员继续面临的一项重大挑战。虽然目前的评价工具有能力对系统的业绩提供可靠的总体概览,但正如最近关于这一专题的研究所显示的那样,在使用这些工具方面仍然缺乏一致性和有效性。目前的传统评估技术往往无法发现数据中较小子集可能发生的差异,也没有能力解释这些差异如何影响总体业绩。在本篇文章中,我们侧重于建议者评价的数据集群概念,对建议者应用的数据集采用邻里评估方法。这种新方法,即以邻里为基础的评价,有助于更好地了解系统中较紧凑的组别中的关键绩效差异,以帮助发现这些差异通常与常规指标不相干且通常平均的弱点。这个新的模块评价层补充了现有的评估机制,并为建议者生态系统提供了多种应用的可能性,例如模型演化测试、欺诈/攻击探测以及主办混合模型设置的可能性。