When interacting with objects through cameras, or pictures, users often have a specific intent. For example, they may want to perform a visual search. With most object detection models relying on image pixels as their sole input, undesired results are not uncommon. Most typically: lack of a high-confidence detection on the object of interest, or detection with a wrong class label. The issue is especially severe when operating capacity-constrained mobile object detectors on-device. In this paper we investigate techniques to modulate mobile detectors to explicitly account for the user intent, expressed as an embedding of a simple query. Compared to standard detectors, query-modulated detectors show superior performance at detecting objects for a given user query. Thanks to large-scale training data synthesized from standard object detection annotations, query-modulated detectors also outperform a specialized referring expression recognition system. Query-modulated detectors can also be trained to simultaneously solve for both localizing a user query and standard detection, even outperforming standard mobile detectors at the canonical COCO task.
翻译:在通过相机或图片与对象进行互动时,用户往往有特定意图。例如,他们可能想要进行视觉搜索。大多数对象检测模型依赖图像像素作为唯一的输入,结果并不罕见。最典型的是:在感兴趣的对象上缺乏高度自信的检测,或用错误的分类标签进行检测。当操作能力受限制的移动物体探测器在装置上操作时,问题特别严重。在本文件中,我们调查调整移动探测器以明确说明用户意图的技术,表现为嵌入简单查询。与标准探测器相比,查询调控探测器显示在为用户查询探测对象时的优异性。由于从标准对象检测说明中合成的大规模培训数据,查询调制探测器也超越了专门引用识别系统。对于用户查询和标准检测,即使是在Canonical CO 任务中,Query-调制探测器也可以同时被训练为本地化用户查询和标准检测而同时解决,甚至超标准移动探测器。