User intent classification is an important task in information retrieval. In this work, we introduce a revised taxonomy of user intent. We take the widely used differentiation between navigational, transactional and informational queries as a starting point, and identify three different sub-classes for the informational queries: instrumental, factual and abstain. The resulting classification of user queries is more fine-grained, reaches a high level of consistency between annotators, and can serve as the basis for an effective automatic classification process. The newly introduced categories help distinguish between types of queries that a retrieval system could act upon, for example by prioritizing different types of results in the ranking.We have used a weak supervision approach based on Snorkel to annotate the ORCAS dataset according to our new user intent taxonomy, utilising established heuristics and keywords to construct rules for the prediction of the intent category. We then present a series of experiments with a variety of machine learning models, using the labels from the weak supervision stage as training data, but find that the results produced by Snorkel are not outperformed by these competing approaches and can be considered state-of-the-art. The advantage of a rule-based approach like Snorkel's is its efficient deployment in an actual system, where intent classification would be executed for every query issued. The resource released with this paper is the ORCAS-I dataset: a labelled version of the ORCAS click-based dataset of Web queries, which provides 18 million connections to 10 million distinct queries.
翻译:用户意图分类是信息检索的一项重要任务。 在这项工作中,我们引入了用户意图的订正分类法。 我们把广泛使用的导航、交易和信息查询的分类法作为起点,并确定了信息查询的三种不同的子类:工具、事实和弃权。 由此对用户查询的分类方法更精细,在批注者之间达到高度的一致性,并可作为有效的自动分类程序的基础。 新引入的分类法有助于区分检索系统可以采取行动的查询类型,例如,通过排列排名中不同类型的结果。 我们使用了基于Snorkel的薄弱监督方法,根据我们新的用户意图分类法,对ORCAS数据集进行注解,并确定了三种不同的分类方法:工具工具:工具使用固定的超链接和关键词,以构建预测意向类别的规则。然后,我们用各种机器学习模型进行一系列实验,使用基于薄弱监督阶段的标签作为培训数据,但发现Snordel产生的结果没有在这些相互竞争的方法上过分级,因此可以认为基于Snockel的系统在实际部署或方向上, 一种不同的数据库工具,在每100万次的分类中可以使用一种不同的数据库。