Multi-modal knowledge graphs (MKGs) include not only the relation triplets, but also related multi-modal auxiliary data (i.e., texts and images), which enhance the diversity of knowledge. However, the natural incompleteness has significantly hindered the applications of MKGs. To tackle the problem, existing studies employ the embedding-based reasoning models to infer the missing knowledge after fusing the multi-modal features. However, the reasoning performance of these methods is limited due to the following problems: (1) ineffective fusion of multi-modal auxiliary features; (2) lack of complex reasoning ability as well as inability to conduct the multi-hop reasoning which is able to infer more missing knowledge. To overcome these problems, we propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning). Specifically, the model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process, based on the features obtained in component (1). The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.
翻译:多模式知识图表(MKGs)不仅包括三重关系,而且还包括相关的多模式辅助数据(即文本和图像),这增加了知识的多样性,然而,自然的不完全性极大地妨碍了MKGs的应用。为解决这一问题,现有研究采用基于嵌入的推理模型来推算多模式特征之后的缺失知识。然而,这些方法的推理性能因以下问题而受到限制:(1) 多模式辅助特征的无效融合;(2) 缺乏复杂的推理能力以及无法进行能够推断更多缺失知识的多模式推理(即文本和图像)。为克服这些问题,我们提议了一个名为MMKGGs的新模式(MMKM-Hop多模式知识图解理学)。具体地说,模型包含以下两个组成部分:(1) 统一的大门保护网络,目的是通过充分关注互动和减少噪音产生有效的多模式互补特征;(2) 补充特征认知强化学习方法,建议通过执行多窗口推理学流程预测缺失要素。(1) 在多模式推理学过程中,根据获得的MKGM-G的状态特征,展示MR工作成果。