Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance.
翻译:模拟推理是人类认知的基础,在各个领域占有重要地位,然而,以往的研究主要侧重于单一模式模拟推理,忽视了结构知识,值得注意的是,认知心理学的研究显示,来自多式联运来源的信息总是带来比单一模式来源更强大的认知转移。为此,我们引入了对知识图进行多模式模拟推理的新任务,这需要在背景知识帮助下建立多模式推理能力。具体地说,我们构建了多模式分析推理数据和多模式知识图MarKG。我们用多模式知识图嵌入和预先培训的变异器基线进行评估,说明拟议任务的潜在挑战。我们进一步提出了由结构绘图理论驱动的新型模型-通识多模式模拟推理框架,可以取得更好的性能。