Tree-based models such as decision trees and random forests (RF) are a cornerstone of modern machine-learning practice. To mitigate overfitting, trees are typically regularized by a variety of techniques that modify their structure (e.g. pruning). We introduce Hierarchical Shrinkage (HS), a post-hoc algorithm that does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors. The amount of shrinkage is controlled by a single regularization parameter and the number of data points in each ancestor. Since HS is a post-hoc method, it is extremely fast, compatible with any tree growing algorithm, and can be used synergistically with other regularization techniques. Extensive experiments over a wide variety of real-world datasets show that HS substantially increases the predictive performance of decision trees, even when used in conjunction with other regularization techniques. Moreover, we find that applying HS to each tree in an RF often improves accuracy, as well as its interpretability by simplifying and stabilizing its decision boundaries and SHAP values. We further explain the success of HS in improving prediction performance by showing its equivalence to ridge regression on a (supervised) basis constructed of decision stumps associated with the internal nodes of a tree. All code and models are released in a full-fledged package available on Github (github.com/csinva/imodels)
翻译:以树为基础的模型,例如决策树和随机森林(RF),是现代机器学习做法的基石。为了减轻过度改造,树木通常通过改变结构的各种技术(例如修剪)加以正规化。我们引入了等级式缩小法(HS),这是一种不会改变树结构的后热算法,而是通过减少对每个节点的预测,使其与祖先的样本方法相结合,使树正规化。缩缩水量由单一的正规化参数和每个祖先中的数据点数控制。由于HS是一种后热方法,它非常快,与任何树生长的算法相容,并且可以与其他正规化技术协同使用。我们对一系列广泛的现实世界数据集的广泛实验表明,HS大大提高了决策树的预测性能,即使与其他正规化技术结合使用。此外,我们发现,将HS应用于每个树中的每棵树往往提高准确性,并通过简化和稳定其决定界限和SHAP值加以解释。我们进一步解释HS在改进HS的精确性模型方面是否成功。我们进一步解释了HS的精确度,而没有通过完全的模型来显示其内部定值的精确性模型。