We study a fundamental transfer learning process from source to target linear regression tasks, including overparameterized settings where there are more learned parameters than data samples. The target task learning is addressed by using its training data together with the parameters previously computed for the source task. We define the target task as a linear regression optimization with a regularization on the distance between the to-be-learned target parameters and the already-learned source parameters. This approach can be also interpreted as adjusting the previously learned source parameters for the purpose of the target task, and in the case of sufficiently related tasks this process can be perceived as fine tuning. We analytically characterize the generalization performance of our transfer learning approach and demonstrate its ability to resolve the peak in generalization errors in double descent phenomena of min-norm solutions to ordinary least squares regression. Moreover, we show that for sufficiently related tasks the optimally tuned transfer learning approach can outperform the optimally tuned ridge regression method, even when the true parameter vector conforms with isotropic Gaussian prior distribution. Namely, we demonstrate that transfer learning can beat the minimum mean square error (MMSE) solution of the individual target task.
翻译:我们研究了一种从源到线性回归任务的基本转移学习过程,包括比数据样本更具有学习程度的参数的超度参数设置。目标任务学习是通过使用其培训数据和先前为源性任务计算的参数来解决的。我们把目标任务定义为线性回归优化,在要学的目标参数和已经学得的来源参数之间的距离上进行正规化。这个方法也可以被解释为为目标任务的目的调整以前学到的源参数,在有足够关联的任务中,这个过程可以被视为微调。我们用分析来描述我们转移学习方法的通用性能,并展示它有能力解决普通最小回归的微调解决方案的双向下沉现象中的一般错误。此外,我们表明,对于充分相关的任务,最佳调整的转移学习方法可以超过最佳调整的山脊回归方法,即使真正的参数矢量与先前的等离子体分布一致。也就是说,我们证明转移学习能够战胜单个任务的最低中值平方差的解决方案。