Conversational search (CS) has recently become a significant focus of the information retrieval (IR) research community. Multiple studies have been conducted which explore the concept of conversational search. Understanding and advancing research in CS requires careful and detailed evaluation. Existing CS studies have been limited to evaluation based on simple user feedback on task completion. We propose a CS evaluation framework which includes multiple dimensions: search experience, knowledge gain, software usability, cognitive load and user experience, based on studies of conversational systems and IR. We introduce these evaluation criteria and propose their use in a framework for the evaluation of CS systems.
翻译:最近,交流搜索已成为信息检索研究界的一个重要重点,已经开展了多项研究,探讨对话搜索的概念。理解和推进CS的研究需要认真和详细的评估。现有的CS研究限于基于简单用户对任务完成情况的反馈的评价。我们提议一个CS评价框架,包括多个方面:搜索经验、知识获取、软件可用性、认知负荷和用户经验,基于对对话系统和IR的研究。我们介绍这些评估标准,并建议在CS系统评价框架内使用这些标准。