Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and flexible datasets for experiments and evaluation. As collecting such datasets is time- and labor-consuming, we propose to use text generation methods to gather datasets. The generator should be trained to generate utterances that belong to the given intent. We explore two approaches to generating task-oriented utterances. In the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training. In the one-shot approach, the model is presented with a single utterance from a test intent. We perform a thorough automatic, and human evaluation of the dataset generated utilizing two proposed approaches. Our results reveal that the attributes of the generated data are close to original test sets, collected via crowd-sourcing.
翻译:意图分类的子任务,如分配转换的稳健性、适应特定用户群体和个性化、外域探测,需要广泛和灵活的数据集来进行实验和评价。由于收集这类数据集耗时费力,我们提议使用文本生成方法来收集数据集。应该对生成者进行培训,以生成属于特定意图的语句。我们探讨了产生任务导向语句的两种方法。在零点法中,该模型经过培训,从可见的意向中生成语句,并被进一步用于生成培训期间看不见的意向的语句。在一发法中,该模型以测试意图的单一语句形式展示。我们利用两种拟议方法对生成的数据集进行彻底的自动和人文评估。我们的结果显示,生成数据的属性接近原始测试组,通过众包采集。