We propose Distribution Embedding Networks (DEN) for classification with small data. In the same spirit of meta-learning, DEN learns from a diverse set of training tasks with the goal to generalize to unseen target tasks. Unlike existing approaches which require the inputs of training and target tasks to have the same dimension with possibly similar distributions, DEN allows training and target tasks to live in heterogeneous input spaces. This is especially useful for tabular-data tasks where labeled data from related tasks are scarce. DEN uses a three-block architecture: a covariate transformation block followed by a distribution embedding block and then a classification block. We provide theoretical insights to show that this architecture allows the embedding and classification blocks to be fixed after pre-training on a diverse set of tasks; only the covariate transformation block with relatively few parameters needs to be fine-tuned for each new task. To facilitate training, we also propose an approach to synthesize binary classification tasks, and demonstrate that DEN outperforms existing methods in a number of synthetic and real tasks in numerical studies.
翻译:我们建议分配嵌入网络(DEN)使用小数据进行分类。本着同样的元学习精神,DEN从一系列不同的培训任务中学习,目标是向看不见的目标任务推广。与要求培训投入和目标任务具有相同层面且分布可能相似的现有方法不同,DEN允许培训和目标任务生活在不同输入空间中。这对于表格数据任务特别有用,因为相关任务中标签的数据很少。DEN使用三块结构:一个共变转换块,然后是分布嵌入块,然后是分类块。我们提供理论见解,以表明这一结构允许在对一系列不同任务进行预先培训后固定嵌入和分类块;只有参数相对较少的共变式转换块,才能为每项新任务进行微调。为了便利培训,我们还提议了一种综合二元分类任务的方法,并表明DEN在数字研究的一些合成和实际任务中超越了现有方法。