In traditional Visual Question Generation (VQG), most images have multiple concepts (e.g. objects and categories) for which a question could be generated, but models are trained to mimic an arbitrary choice of concept as given in their training data. This makes training difficult and also poses issues for evaluation -- multiple valid questions exist for most images but only one or a few are captured by the human references. We present Guiding Visual Question Generation - a variant of VQG which conditions the question generator on categorical information based on expectations on the type of question and the objects it should explore. We propose two variants: (i) an explicitly guided model that enables an actor (human or automated) to select which objects and categories to generate a question for; and (ii) an implicitly guided model that learns which objects and categories to condition on, based on discrete latent variables. The proposed models are evaluated on an answer-category augmented VQA dataset and our quantitative results show a substantial improvement over the current state of the art (over 9 BLEU-4 increase). Human evaluation validates that guidance helps the generation of questions that are grammatically coherent and relevant to the given image and objects.
翻译:在传统的视觉问题生成(VQG)中,大多数图像有多种概念(例如对象和类别),可以产生问题,但模型经过培训,可以模仿其培训数据中给出的任意选择概念。这给培训造成困难,并提出了评估问题 -- -- 大多数图像存在多种有效的问题,但人类参考文献只记录其中的一或几张。我们介绍了指导视觉问题生成 -- -- VQG的一个变式,根据对问题类型和应探索对象的预期,使问题生成者以绝对信息为条件。我们提出了两个变式:(一) 明确指导模型,使一个行为者(人或自动)能够选择哪些对象和类别来产生问题;以及(二) 隐含指导模型,根据离散的潜伏变量学习哪些对象和类别的条件。我们用一个答案类别扩大VQA数据集对拟议模型进行了评估,我们的定量结果显示,与艺术的现状相比,(超过9个BLEU-4增加)有很大的改进。我们提出了两种变式。我们提出了两种变式:(一)明确指导,使一个行为者(人或自动)能够选择哪些对象和类别一致和与给特定对象和相关的问题。