From Shapley Values to Generalized Additive Models and back

In explainable machine learning, local post-hoc explanation algorithms and inherently interpretable models are often seen as competing approaches. In this work, we offer a partial reconciliation between these two approaches by showing that Shapley Values correspond to Generalized Additive Models (GAMs). We introduce $n$-Shapley Values, a parametric family of local post-hoc explanation algorithms that explain individual predictions with interaction terms up to order $n$. By varying the parameter $n$, these explanations cover the entire range from Shapley Values up to a uniquely determined decomposition of the function that we attempt to explain. The relationship between $n$-Shapley Values and this decomposition offers a functionally-grounded characterization of Shapley Values, and highlights the limitations of these explanations. We then show that $n$-Shapley Values recover GAMs with interaction terms up to order $n$, which implies that the original Shapely Values recover GAMs without interaction terms. Taken together, our results offer a precise characterization of Shapley Values as they are being used in explainable machine learning. Python code to estimate $n$-Shapley Values and replicate the results in this paper is available at \url{https://github.com/tml-tuebingen/nshap}.