A Practical & Unified Notation for Information-Theoretic Quantities in ML

Information theory is of importance to machine learning, but the notation for information-theoretic quantities is sometimes opaque. The right notation can convey valuable intuitions and concisely express new ideas. We propose such a notation for machine learning users and expand it to include information-theoretic quantities between observed outcomes (events) and random variables. To demonstrate the value of our notation, first, we apply it to elegantly prove a version of Stirling's approximation for binomial coefficients mentioned by MacKay. Second, we apply the notation to a popular information-theoretic acquisition function in Bayesian active learning which selects the most informative (unlabelled) samples to be labelled by an expert and extend this acquisition function to the core-set problem, which consists of selecting the most informative samples \emph{given} the labels.