In supervised learning, training and test datasets are often sampled from distinct distributions. Domain adaptation techniques are thus required. Covariate shift adaptation yields good generalization performance when domains differ only by the marginal distribution of features. Covariate shift adaptation is usually implemented using importance weighting, which may fail, according to common wisdom, due to small effective sample sizes (ESS). Previous research argues this scenario is more common in high-dimensional settings. However, how effective sample size, dimensionality, and model performance/generalization are formally related in supervised learning, considering the context of covariate shift adaptation, is still somewhat obscure in the literature. Thus, a main challenge is presenting a unified theory connecting those points. Hence, in this paper, we focus on building a unified view connecting the ESS, data dimensionality, and generalization in the context of covariate shift adaptation. Moreover, we also demonstrate how dimensionality reduction or feature selection can increase the ESS, and argue that our results support dimensionality reduction before covariate shift adaptation as a good practice.
翻译:在受监督的学习、培训和测试数据集中,往往从不同的分布中取样。因此,需要从不同的分布中挑选出不同的适应技术。在区域因地貌的边际分布而不同时,可变的适应产生良好的概括性效果。常变的适应通常使用重要性加权法实施,而根据共同的智慧,由于有效抽样规模小,这种加权法可能失败。以前的研究认为,这种假设在高维环境中更为常见。然而,考虑到共变式转移适应的背景,在受监督的学习中,有效的抽样规模、维度和模型性能/一般化如何正式联系起来,在文献中仍然有些模糊。因此,一项主要的挑战是提出一种将这些点连接起来的统一理论。因此,在本文中,我们侧重于在共变式转移适应的背景下,建立统一的观点,将ESS、数据维度和一般化联系起来。此外,我们还表明,在共变式转移适应方面减少或特征选择如何能增加ESS,并论证我们的结果支持在共变式调整适应之前减少维度。