A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method.
翻译:提出了一种称作“注意随机森林”的新办法,并修改了对随机森林的注意机制,以进行回归和分类。拟议“注意”模式的主要想法是,以特定的方式将关注权重与可训练参数分配给决策树。加权取决于一个实例之间的距离,该实例属于一棵树的相应叶,而实例则属于同一叶中。这一想法来自以RF的形式表示Nadaraya-Watson内核回归。提出了对一般方法的三项修改。第一个是应用Huber的污染模型,并通过解决二次和第三次修改使用梯度算法计算可训练参数。数字式实验以各种回归和分类数据集的形式说明了拟议方法。