Image captioning (IC) systems, which automatically generate a text description of the salient objects in an image (real or synthetic), have seen great progress over the past few years due to the development of deep neural networks. IC plays an indispensable role in human society, for example, labeling massive photos for scientific studies and assisting visually-impaired people in perceiving the world. However, even the top-notch IC systems, such as Microsoft Azure Cognitive Services and IBM Image Caption Generator, may return incorrect results, leading to the omission of important objects, deep misunderstanding, and threats to personal safety. To address this problem, we propose MetaIC, the \textit{first} metamorphic testing approach to validate IC systems. Our core idea is that the object names should exhibit directional changes after object insertion. Specifically, MetaIC (1) extracts objects from existing images to construct an object corpus; (2) inserts an object into an image via novel object resizing and location tuning algorithms; and (3) reports image pairs whose captions do not exhibit differences in an expected way. In our evaluation, we use MetaIC to test one widely-adopted image captioning API and five state-of-the-art (SOTA) image captioning models. Using 1,000 seeds, MetaIC successfully reports 16,825 erroneous issues with high precision (84.9\%-98.4\%). There are three kinds of errors: misclassification, omission, and incorrect quantity. We visualize the errors reported by MetaIC, which shows that flexible overlapping setting facilitates IC testing by increasing and diversifying the reported errors. In addition, MetaIC can be further generalized to detect label errors in the training dataset, which has successfully detected 151 incorrect labels in MS COCO Caption, a standard dataset in image captioning.
翻译:图像标题 (IC) 系统自动生成图像( 真实的或合成的) 突出对象的文字描述, 过去几年里, 由于深层神经网络的发展, 该系统取得了巨大的进步。 IC 在人类社会中发挥了不可或缺的作用, 例如, 为科学研究贴上大片照片, 并帮助视视障者感知世界。 但是, 即使顶尖的IC系统, 如 微软 Azure 识别服务 和 IBM 图像描述生成器等, 也可能会返回错误的结果, 导致重要对象的遗漏、 深刻的误解和对人身安全的威胁。 为了解决这个问题, 我们提议MetAIIC, 将 & textit{First 变异性测试方法用于验证 IC 系统。 我们的核心思想是, 将对象名称标为在对象插入后显示方向变化变化。 具体来说, MetAIIC (1) 从现有图像中提取对象来构建一个天体; (2) 通过新对象调整和位置校正算算法, 以及(3) 报告图像配对, 其标题不会在预期的方式出现差异。 在评估中, 我们使用MetIC 284 显示 高级 数据 版本 显示 版本 5 版本 。