Multi-label image recognition with partial labels (MLR-PL), in which some labels are known while others are unknown for each image, may greatly reduce the cost of annotation and thus facilitate large-scale MLR. We find that strong semantic correlations exist within each image and across different images, and these correlations can help transfer the knowledge possessed by the known labels to retrieve the unknown labels and thus improve the performance of the MLR-PL task (see Figure 1). In this work, we propose a novel heterogeneous semantic transfer (HST) framework that consists of two complementary transfer modules that explore both within-image and cross-image semantic correlations to transfer the knowledge possessed by known labels to generate pseudo labels for the unknown labels. Specifically, an intra-image semantic transfer (IST) module learns an image-specific label co-occurrence matrix for each image and maps the known labels to complement the unknown labels based on these matrices. Additionally, a cross-image transfer (CST) module learns category-specific feature-prototype similarities and then helps complement the unknown labels that have high degrees of similarity with the corresponding prototypes. Finally, both the known and generated pseudo labels are used to train MLR models. Extensive experiments conducted on the Microsoft COCO, Visual Genome, and Pascal VOC 2007 datasets show that the proposed HST framework achieves superior performance to that of current state-of-the-art algorithms. Specifically, it obtains mean average precision (mAP) improvements of 1.4%, 3.3%, and 0.4% on the three datasets over the results of the best-performing previously developed algorithm.
翻译:多标签图像识别( MLR-PL) 部分标签( MLR-PL) 部分标签( MLR-PL-PL), 部分标签为已知, 而其他标签为未知) 的多标签图像识别(MLR-PL), 可能会大大降低批注成本, 从而便利大型 MLR 。 我们发现, 在每个图像内部和不同图像之间都存在强烈的语义相关关系, 这些关联有助于传输已知标签拥有的知识, 以检索未知标签, 从而改善 MLR- PL- PL任务的性能(见图1) 。 在这项工作中, 我们提议建立一个新颖的混杂语种语义传输(HST) 框架, 包括两个互补的传输模块, 探索图像内和交叉图像内和跨图像性平面性平比的数学相关关系, 从而补充了已知的2007年SLOV-ROD的高级性价比 。