Maximal Causes for Exponential Family Observables

Latent variable models represent observed variables as parameterized functions of a set of latent variables. Examples are factor analysis or probabilistic sparse coding which assume weighted linear summations to determine the mean of Gaussian distribution for the observables. However, in many cases observables do not follow a normal distribution, and a linear summation of latents is often at odds with non-Gaussian observables (e.g., means of the Bernoulli distribution have to lie in the unit interval). Furthermore, the assumption of a linear summation model may (for many types of data) not be closely aligned with the true data generation process even for Gaussian observables. Alternative superposition models (i.e., alternative links between latents and observables) have therefore been investigated repeatedly. Here we show that using the maximization instead of summation to link latents to observables allows for the derivation of a very general and concise set of parameter update equations. Concretely, we derive a set of update equations that has the same functional form for all distributions of the exponential family. Our results consequently provide directly applicable learning equations for commonly as well as for unusually distributed data. We numerically verify our analytical results assuming standard Gaussian, Gamma, Poisson, Bernoulli and Exponential distributions. We point to some potential applications by providing different experiments on the learning of variance structure, noise type estimation, and denoising.