Deep neural networks have achieved promising results in automatic image captioning due to their effective representation learning and context-based content generation capabilities. As a prominent type of deep features used in many of the recent image captioning methods, the well-known bottomup features provide a detailed representation of different objects of the image in comparison with the feature maps directly extracted from the raw image. However, the lack of high-level semantic information about the relationships between these objects is an important drawback of bottom-up features, despite their expensive and resource-demanding extraction procedure. To take advantage of visual relationships in caption generation, this paper proposes a deep neural network architecture for image captioning based on fusing the visual relationships information extracted from an image's scene graph with the spatial feature maps of the image. A multi-modal reward function is then introduced for deep reinforcement learning of the proposed network using a combination of language and vision similarities in a common embedding space. The results of extensive experimentation on the MSCOCO dataset show the effectiveness of using visual relationships in the proposed captioning method. Moreover, the results clearly indicate that the proposed multi-modal reward in deep reinforcement learning leads to better model optimization, outperforming several state-of-the-art image captioning algorithms, while using light and easy to extract image features. A detailed experimental study of the components constituting the proposed method is also presented.
翻译:深度神经网络由于其有效的表示学习和基于上下文的内容生成能力,在自动图像字幕生成方面取得了有前途的结果。作为许多最近的图像字幕方法中使用的杰出深度特征的一种,众所周知的 bottom-up 特征相比直接从原始图像中提取的特征映射,提供了有关图像中不同对象的详细表示。然而,尽管它们的昂贵和资源需求量很大的特征提取过程,但它们缺乏有关这些对象之间关系的高级语义信息,这是它们的一个重要缺点。为了利用字幕生成中的视觉关系,本文提出了一种基于从图像场景图中提取的视觉关系信息与图像的空间特征映射融合的深度神经网络架构。然后,引入了一个多模态奖励函数,用于在常见嵌入空间中使用语言和视觉相似性的组合进行所提出网络的深度强化学习。广泛实验的结果表明,使用视觉关系在所提出的字幕生成方法中的有效性。此外,结果明确表明,深度强化学习中所提出的多模态奖励导致更好的模型优化,优于几种最先进的图像字幕算法,同时使用轻便且易于提取的图像特征。还提供了所提出方法的组成部分的详细实验研究。