Representing probability distributions by the gradient of their density functions has proven effective in modeling a wide range of continuous data modalities. However, this representation is not applicable in discrete domains where the gradient is undefined. To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings. Given a predefined neighborhood structure, the Concrete score of any input is defined by the rate of change of the probabilities with respect to local directional changes of the input. This formulation allows us to recover the (Stein) score in continuous domains when measuring such changes by the Euclidean distance, while using the Manhattan distance leads to our novel score function in discrete domains. Finally, we introduce a new framework to learn such scores from samples called Concrete Score Matching (CSM), and propose an efficient training objective to scale our approach to high dimensions. Empirically, we demonstrate the efficacy of CSM on density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets, and demonstrate that it performs favorably relative to existing baselines for modeling discrete data.
翻译:以其密度函数的梯度表示概率分布,已证明在模拟一系列广泛的连续数据模式方面是有效的。然而,这种表示方式不适用于梯度未定义的离散域。 为此,我们提议了一个类似评分函数,称为“Concrete 评分”,这是对离散设置的(Stein)评分的概括化。鉴于预先定义的周边结构,任何输入的分数的大小都取决于投入的本地方向变化的概率变化率的变化率。这种配方使我们能够在用Euclidean距离测量这些变化时,在连续域中恢复(stein)评分,而同时使用曼哈顿距离则导致我们在离散域的新评分功能。最后,我们引入了一个新的框架,从称为Cecomm分匹配(CSM)的样本中学习这种评分,并提议一个有效的培训目标,以将我们的方法推广到高维度。我们很生地展示了CSMM在合成、表和高维图像数据集的密度估计任务方面的效力,并表明它与现有的模型化基线相对良好。