Bit Allocation using Optimization

In this paper, we consider the problem of bit allocation in Neural Video Compression (NVC). First, we reveal a fundamental relationship between bit allocation in NVC and Semi-Amortized Variational Inference (SAVI) with Group-of-Picture (GoP) level likelihood. Specifically, we show that SAVI with GoP-level likelihood is equivalent to optimal pixel-level bit allocation with precise rate \& quality dependency model. Based on this equivalence, we establish a new paradigm of bit allocation using SAVI. Different from previous bit allocation methods, our approach requires no empirical model and is thus optimal. Moreover, as the original SAVI using gradient ascent only applies to fully factorized latent, we extend the SAVI to dependent latent such as NVC by recursively applying back-propagating through gradient ascent. Finally, we propose a tractable approximation to this algorithm for practical implementation. Our algorithm can be applied to scenarios where performance outweights encoding speed, and serves as an empirical bound on the R-D performance of bit allocation. Experimental results show that our extension of SAVI outperforms original SAVI on dependent latent, and current state-of-the-art bit allocation algorithms still have a room of $\approx 0.5$ dB PSNR to improve compared with ours. Code is available in supplementary materials.