Learning energy-based models (EBMs) is known to be difficult especially on discrete data where gradient-based learning strategies cannot be applied directly. Although ratio matching is a sound method to learn discrete EBMs, it suffers from expensive computation and excessive memory requirements, thereby resulting in difficulties in learning EBMs on high-dimensional data. Motivated by these limitations, in this study, we propose ratio matching with gradient-guided importance sampling (RMwGGIS). Particularly, we use the gradient of the energy function w.r.t. the discrete data space to approximately construct the provably optimal proposal distribution, which is subsequently used by importance sampling to efficiently estimate the original ratio matching objective. We perform experiments on density modeling over synthetic discrete data, graph generation, and training Ising models to evaluate our proposed method. The experimental results demonstrate that our method can significantly alleviate the limitations of ratio matching, perform more effectively in practice, and scale to high-dimensional problems. Our implementation is available at https://github.com/divelab/RMwGGIS.
翻译:已知学习能源模型(EBMS)很难,特别是对于不能直接应用梯度学习战略的离散数据而言。虽然比对是学习离散EBM的健全方法,但因计算费用昂贵和记忆要求过多,导致难以在高维数据方面学习EBM(EBMS),受这些局限性的驱使,我们在本研究报告中提议与梯度引导重要性抽样(RMwGGIS)匹配比,特别是,我们使用离散数据空间的梯度,以大致构建可察觉的最佳建议分布,随后通过重要取样来有效估计原始比对比目标。我们对合成离散数据进行密度建模试验、图生成和培训Ising模型来评价我们提议的方法。实验结果表明,我们的方法可以大大减轻比对比的局限性,在实际中更有效地发挥作用,在高维度问题上推广。我们的实施方法可在https://github.com/divelab/RMGGGGIS查阅。</s>