This paper compares the performances of three supervised machine learning algorithms in terms of predictive ability and model interpretation on structured or tabular data. The algorithms considered were scikit-learn implementations of extreme gradient boosting machines (XGB) and random forests (RFs), and feedforward neural networks (FFNNs) from TensorFlow. The paper is organized in a findings-based manner, with each section providing general conclusions supported by empirical results from simulation studies that cover a wide range of model complexity and correlation structures among predictors. We considered both continuous and binary responses of different sample sizes. Overall, XGB and FFNNs were competitive, with FFNNs showing better performance in smooth models and tree-based boosting algorithms performing better in non-smooth models. This conclusion held generally for predictive performance, identification of important variables, and determining correct input-output relationships as measured by partial dependence plots (PDPs). FFNNs generally had less over-fitting, as measured by the difference in performance between training and testing datasets. However, the difference with XGB was often small. RFs did not perform well in general, confirming the findings in the literature. All models exhibited different degrees of bias seen in PDPs, but the bias was especially problematic for RFs. The extent of the biases varied with correlation among predictors, response type, and data set sample size. In general, tree-based models tended to over-regularize the fitted model in the tails of predictor distributions. Finally, as to be expected, performances were better for continuous responses compared to binary data and with larger samples.
翻译:本文比较了三个受监督的机器学习算法在预测能力和对结构或表格数据进行模型解释方面的绩效; 所考虑的算法是:在预测能力方面三个受监督的机器学习算法的性能,以及在结构或表格数据方面的模型解释; 所考虑的算法是:对极端梯度助推机(XGB)和随机森林(RFs)的Scikit-learn 实施Scikit-learn 执行,以及TensorFlow的Forward神经网络(FFNNes)的forforfor 。 本文以基于结果的方式编排,每一部分依赖图(PDPs)的模拟研究结果提供了一般性结论,其中涵盖了各种模型的复杂性和相关性结构。 总的来说, XGB 和FFNes 具有竞争力,但在光滑动模型和树基助推推推算法模型的性能方面表现较好。 在一般预测性能图中,与一般的准确性模型相比,与总性能的准确性值之间的差异,在总的准确性模型中,其结果与一般性能与一般性能的准确性判断性值之间并非。