Unpicking Data at the Seams: Understanding Disentanglement in VAEs

Disentanglement, or identifying statistically independent factors of the data, is relevant to much of machine learning, from controlled data generation and robust classification to efficient encoding and improving our understanding of the data itself. Disentanglement arises in several generative paradigms including Variational Autoencoders (VAEs), Generative Adversarial Networks and diffusion models. Recent progress has been made in understanding disentanglement in VAEs, where a choice of diagonal posterior covariance matrices is shown to promote mutual orthogonality between columns of the decoder's Jacobian. We build on this to show how such orthogonality, a geometric property, translates to disentanglement, a statistical property, furthering our understanding of how a VAE identifies independent components of, or disentangles, the data.