Transformers have achieved remarkable performance in widespread fields, including natural language processing, computer vision and graph mining. However, in the knowledge graph representation, where translational distance paradigm dominates this area, vanilla Transformer architectures have not yielded promising improvements. Note that vanilla Transformer architectures struggle to capture the intrinsically semantic and structural information of knowledge graphs and can hardly scale to long-distance neighbors due to quadratic dependency. To this end, we propose a new variant of Transformer for knowledge graph representation dubbed Relphormer. Specifically, we introduce Triple2Seq which can dynamically sample contextualized sub-graph sequences as the input of the Transformer to alleviate the scalability issue. We then propose a novel structure-enhanced self-attention mechanism to encode the relational information and keep the globally semantic information among sub-graphs. Moreover, we propose masked knowledge modeling as a new paradigm for knowledge graph representation learning to unify different link prediction tasks. Experimental results show that our approach can obtain better performance on benchmark datasets compared with baselines.
翻译:在包括自然语言处理、计算机视觉和图解挖掘在内的广泛领域,变异器取得了显著的绩效。然而,在知识图示中,翻译距离模式占了该领域的主导位置,但香草变异器结构没有带来有希望的改进。请注意,香草变异器结构在努力捕捉知识图中固有的语义和结构信息,由于四面形依赖性,很难将其推广到长距离邻居。为此,我们提出了一个新的变异器,用于知识图解代表,代号为Relphormer。具体地说,我们引入了Triple2Seq,它能动态地样样样地采样子子图序列,作为变异器的输入,以缓解可缩放问题。我们随后提出了一个新的结构强化自我注意机制,以编码关系信息,并将全球的语义信息保留在子图中。此外,我们提出了隐蔽知识模型,作为知识图表学习的新模式,以统一不同链接的预测任务。实验结果表明,我们的方法可以在基准数据集上取得比基线更好的业绩。