The goal of supervised representation learning is to construct effective data representations for prediction. Among all the characteristics of an ideal nonparametric representation of high-dimensional complex data, sufficiency, low dimensionality and disentanglement are some of the most essential ones. We propose a deep dimension reduction approach to learning representations with these characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We formulate the ideal representation learning task as that of finding a nonparametric representation that minimizes an objective function characterizing conditional independence and promoting disentanglement at the population level. We then estimate the target representation at the sample level nonparametrically using deep neural networks. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero. Our extensive numerical experiments using simulated and real benchmark data demonstrate that the proposed methods have better performance than several existing dimension reduction methods and the standard deep learning models in the context of classification and regression.
翻译:监督代表制学习的目的是为预测建立有效的数据表示方式; 高维复杂数据、充足性、低维度和分解等理想的非参数表示方式的所有特点都是一些最基本的特征; 我们提议了一种深度减少维度的方法,以了解具有这些特征的表述方式; 提议的方法是对充分的减少维度方法进行非对称的概括化; 我们制定了理想的代表性学习任务,以找到一种非对称的表示方式,以最大限度地减少有条件独立这一客观功能的特征,并促进人口层面的分解; 然后利用深神经网络对抽样层的目标表示方式进行非对称性估计; 我们表明,估计的深度非参数表示方式是一致的,因为其超重风险将集中到零。 我们利用模拟和实际基准数据进行的大量数字实验表明,在分类和回归方面,拟议方法的性能优于现有的若干减少维度方法和标准深层学习模型。