Sliced mutual information (SMI) is defined as an average of mutual information (MI) terms between one-dimensional random projections of the random variables. It serves as a surrogate measure of dependence to classic MI that preserves many of its properties but is more scalable to high dimensions. However, a quantitative characterization of how SMI itself and estimation rates thereof depend on the ambient dimension, which is crucial to the understanding of scalability, remain obscure. This works extends the original SMI definition to $k$-SMI, which considers projections to $k$-dimensional subspaces, and provides a multifaceted account on its dependence on dimension. Using a new result on the continuity of differential entropy in the 2-Wasserstein metric, we derive sharp bounds on the error of Monte Carlo (MC)-based estimates of $k$-SMI, with explicit dependence on $k$ and the ambient dimension, revealing their interplay with the number of samples. We then combine the MC integrator with the neural estimation framework to provide an end-to-end $k$-SMI estimator, for which optimal convergence rates are established. We also explore asymptotics of the population $k$-SMI as dimension grows, providing Gaussian approximation results with a residual that decays under appropriate moment bounds. Our theory is validated with numerical experiments and is applied to sliced InfoGAN, which altogether provide a comprehensive quantitative account of the scalability question of $k$-SMI, including SMI as a special case when $k=1$.
翻译:Sliced 相互信息(SMI)的定义是随机变量的一维随机预测之间的相互信息平均值(MI),它是对传统的MI的替代度,它保存着它的许多属性,但更可伸缩到高维。然而,SMI本身及其估计率的定量特征如何取决于环境层面,这对理解可缩放性至关重要。这把最初SMI的定义扩大到美元-SMI,它考虑到对美元维次空间的预测,并提供了其对维度依赖度的多方面说明。我们利用对2-Wasserstein 公标中差异英特罗比的连续性的新结果,对基于Monte Carlo(MC)的美元-SMI估计数及其估计率的误差进行了精确的分界,明确依赖美元和环境维度,暴露了它们与样本数量之间的相互作用。我们随后将MMC Introgator与神经元的量估量框架结合起来,以提供最终至美元Salimaimal MI 的精度理论, 其最佳趋近率在Sal-Gal-I 的精确度上, 提供了我们Slimal-Ialalal-exal-exal-exal 的精确度的精确度,作为正值的精确的精确度, 。