Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks. The code is available at https://github.com/Linear95/TC-estimation.
翻译:总体相关性(TC)是衡量多种随机变量之间统计依赖性的信息理论中的一个基本概念。最近,TC显示,它作为许多学习任务的一个常规化因素,具有明显的效力,需要共同尽量减少或最大限度地扩大多种潜在嵌入层之间的相互关系。然而,计算精确的TC值具有挑战性,特别是当嵌入变量的封闭式分布不明时。在本文件中,我们引入了一个统一框架,以估计与基于抽样的相互信息估计员(MI)的总体相关性值。更具体地说,我们发现TC和MI之间的关系,并提出将TC解入MI术语的两种计算路径(相似和类似线),以将TC解入MI术语。随着每个MI术语被捆绑起来,可以成功地估算TC值。此外,我们提供关于拟议的TC估计员统计一致性的理论分析。实验既针对合成情景,也针对现实世界情景,我们的估计员们在那里展示了所有TC估计、最小化和最大化任务的有效性。该代码可在https://github.com/Linear95/TC-stimation上查阅。