The task of image captioning aims to generate captions directly from images via the automatically learned cross-modal generator. To build a well-performing generator, existing approaches usually need a large number of described images, which requires a huge effects on manual labeling. However, in real-world applications, a more general scenario is that we only have limited amount of described images and a large number of undescribed images. Therefore, a resulting challenge is how to effectively combine the undescribed images into the learning of cross-modal generator. To solve this problem, we propose a novel image captioning method by exploiting the Cross-modal Prediction and Relation Consistency (CPRC), which aims to utilize the raw image input to constrain the generated sentence in the commonly semantic space. In detail, considering that the heterogeneous gap between modalities always leads to the supervision difficulty of using the global embedding directly, CPRC turns to transform both the raw image and corresponding generated sentence into the shared semantic space, and measure the generated sentence from two aspects: 1) Prediction consistency. CPRC utilizes the prediction of raw image as soft label to distill useful supervision for the generated sentence, rather than employing the traditional pseudo labeling; 2) Relation consistency. CPRC develops a novel relation consistency between augmented images and corresponding generated sentences to retain the important relational knowledge. In result, CPRC supervises the generated sentence from both the informativeness and representativeness perspectives, and can reasonably use the undescribed images to learn a more effective generator under the semi-supervised scenario.
翻译:图像字幕的任务旨在通过自动学习的跨模式生成器直接从图像中产生字幕。 为了构建一个功能良好的生成器, 现有方法通常需要大量描述图像, 这需要对手工标签产生巨大影响。 然而, 在现实世界应用中, 更普遍的假设是, 我们只有数量有限的描述图像和大量未描述的图像。 因此, 由此产生的挑战是如何将未描述的图像有效地结合到跨模式生成器的学习中。 为了解决这个问题, 我们提出一种新的图像字幕说明方法, 利用跨模式预测和关联生成器, 目的是利用原始图像输入来限制通常语义空间中生成的句子。 详细而言, 考虑到模式之间的差异总是导致监督困难, CPRC转而将原始图像和相应生成的句子转化为共同的语义生成空间, 从两个方面衡量生成的句子:(1) 可预测性一致性。 CPRC 利用原始图像预测作为软标签和关联度的原始图像描述值, 利用传统的监管关系, 提高排序, 而不是更新排序 。