Revisiting Classical Multiclass Linear Discriminant Analysis with a Novel Prototype-based Interpretable Solution

Linear discriminant analysis (LDA) is a fundamental method for feature extraction and dimensionality reduction. Despite having many variants, classical LDA has its importance, as it is a keystone in human knowledge about pattern recognition. For a dataset containing $C$ clusters, the classical solution to LDA extracts at most $C-1$ features. In this paper, we introduce a novel solution to classical LDA, called LDA++, that yields $C$ features, each one interpretable as measuring similarity to one cluster. This novel solution bridges between dimensionality reduction and multiclass classification. Specifically, we prove that, under some mild conditions, the optimal weights of a linear multiclass classifier for homoscedastic Gaussian data also make an optimal solution to LDA. In addition, this novel interpretable solution reveals some new facts about LDA and its relation with PCA. We provide a complete numerical solution for our novel method, covering the cases 1) when the scatter matrices can be constructed explicitly, 2) when constructing the scatter matrices is infeasible, and 3) the kernel extension. The code is available at https://github.com/k-ghiasi/LDA-plus-plus.