In this work, we perform an extensive investigation of two state-of-the-art (SotA) methods for the task of Entity Alignment in Knowledge Graphs. Therefore, we first carefully examine the benchmarking process and identify several shortcomings, which make the results reported in the original works not always comparable. Furthermore, we suspect that it is a common practice in the community to make the hyperparameter optimization directly on a test set, reducing the informative value of reported performance. Thus, we select a representative sample of benchmarking datasets and describe their properties. We also examine different initializations for entity representations since they are a decisive factor for model performance. Furthermore, we use a shared train/validation/test split for a fair evaluation setting in which we evaluate all methods on all datasets. In our evaluation, we make several interesting findings. While we observe that most of the time SotA approaches perform better than baselines, they have difficulties when the dataset contains noise, which is the case in most real-life applications. Moreover, we find out in our ablation study that often different features of SotA methods are crucial for good performance than previously assumed. The code is available at https://github.com/mberr/ea-sota-comparison.
翻译:在这项工作中,我们为“知识图表中的实体协调”的任务对两种最先进的方法进行了广泛的调查。因此,我们首先仔细审查基准制定过程,并找出若干缺点,使原始作品中报告的结果并非总能比较。此外,我们怀疑,使超光谱优化直接在一个测试组上进行,从而降低所报告业绩的信息价值,是社区的一个常见做法。因此,我们选择了一个具有代表性的基准数据集样本,并描述其属性。我们还检查了实体代表的不同初始化,因为它们是模型性能的决定性因素。此外,我们使用一个共享的火车/验证/测试拆分来进行公平的评估,用以评估所有数据集的所有方法。在我们的评估中,我们得出若干有趣的结论。虽然我们注意到,多数时间SotA方法比基线要好,但在数据集含有噪音时它们就会有困难,这是大多数真实应用中的情况。此外,我们从我们的模拟研究中发现,SotA方法的不同特征往往对良好的性能至关重要。代码可以在 http://commas/commartari-tomas。