In this paper we analyse and improve integer discrete flows for lossless compression. Integer discrete flows are a recently proposed class of models that learn invertible transformations for integer-valued random variables. Their discrete nature makes them particularly suitable for lossless compression with entropy coding schemes. We start by investigating a recent theoretical claim that states that invertible flows for discrete random variables are less flexible than their continuous counterparts. We demonstrate with a proof that this claim does not hold for integer discrete flows due to the embedding of data with finite support into the countably infinite integer lattice. Furthermore, we zoom in on the effect of gradient bias due to the straight-through estimator in integer discrete flows, and demonstrate that its influence is highly dependent on architecture choices and less prominent than previously thought. Finally, we show how different architecture modifications improve the performance of this model class for lossless compression, and that they also enable more efficient compression: a model with half the number of flow layers performs on par with or better than the original integer discrete flow model.
翻译:在本文中,我们分析并改进了用于无损压缩的整数离散流。 整数离散流是最近提出的一组模型,它们学习了对整数估值随机变量的不可逆变换。 它们的离散性质使得它们特别适合使用整数编码方法进行无损压缩。 我们首先调查最近的一项理论性主张,其中指出离散随机变量的可视流动比对等的连续随机变量的灵活程度要低。 我们用一个证据来证明,由于将有限的支持数据嵌入可计算到可计算到无限整数的整数拉蒂中,这一主张不能维持整数离散流动。 此外,我们放大了由于整数离散流动中直通估计值造成的梯度偏差的影响,并表明其影响高度取决于结构选择,而且比以前想象的要小。 最后,我们展示了不同的结构修改如何改善这个模型类的无损压缩性能,而且它们也使得更有效率的压缩: 一种模型,其流层数的一半与原整数不同或更好。