User simulation has been a cost-effective technique for evaluating conversational recommender systems. However, building a human-like simulator is still an open challenge. In this work, we focus on how users reformulate their utterances when a conversational agent fails to understand them. First, we perform a user study, involving five conversational agents across different domains, to identify common reformulation types and their transition relationships. A common pattern that emerges is that persistent users would first try to rephrase, then simplify, before giving up. Next, to incorporate the observed reformulation behavior in a user simulator, we introduce the task of reformulation sequence generation: to generate a sequence of reformulated utterances with a given intent (rephrase or simplify). We develop methods by extending transformer models guided by the reformulation type and perform further filtering based on estimated reading difficulty. We demonstrate the effectiveness of our approach using both automatic and human evaluation.
翻译:用户模拟是评价对话建议系统的一种具有成本效益的技术。 然而, 建立一个像人一样的模拟器仍是一个公开的挑战。 在这项工作中, 我们侧重于用户在对话代理器无法理解它们时如何重新表述它们的话语。 首先, 我们进行用户研究, 涉及不同领域的五个对话代理器, 以查明共同的重整类型及其过渡关系。 一个常见的模式是, 坚持的用户在放弃之前先尝试重新措辞, 然后简化。 下一步, 将观察到的重整行为纳入用户模拟器, 我们引入重订序列生成的任务: 生成一个带有特定意图( 重拟或简化) 的重订语的序列。 我们开发方法, 扩展由重整型号指导的变压器模型, 并根据估计的阅读困难进行进一步的过滤。 我们用自动和人文评估来展示我们的方法的有效性 。