Transfer learning aims to improve the performance of target tasks by transferring knowledge acquired in source tasks. The standard approach is pre-training followed by fine-tuning or linear probing. Especially, selecting a proper source domain for a specific target domain under predefined tasks is crucial for improving efficiency and effectiveness. It is conventional to solve this problem via estimating transferability. However, existing methods can not reach a trade-off between performance and cost. To comprehensively evaluate estimation methods, we summarize three properties: stability, reliability and efficiency. Building upon them, we propose Principal Gradient Expectation(PGE), a simple yet effective method for assessing transferability. Specifically, we calculate the gradient over each weight unit multiple times with a restart scheme, and then we compute the expectation of all gradients. Finally, the transferability between the source and target is estimated by computing the gap of normalized principal gradients. Extensive experiments show that the proposed metric is superior to state-of-the-art methods on all properties.
翻译:转让学习的目的是通过转让源任务获得的知识来改进目标任务的业绩。标准做法是培训前,然后进行微调或线性调查。特别是,根据预先确定的任务为特定目标领域选择一个适当的源域对于提高效率和效力至关重要。通常的做法是通过估计可转让性来解决这个问题。但是,现有方法无法在业绩和成本之间达成平衡。为了全面评估估算方法,我们总结了三种属性:稳定性、可靠性和效率。我们在此基础上提议了“渐进期望”原则(PGE),这是评估可转让性的一个简单而有效的方法。具体地说,我们用重新启动计划计算每个重量单位的梯度,多次计算每个重量单位的梯度,然后计算所有梯度的预期值。最后,源和目标之间的可转移性是通过计算正常本位梯度差距来估计的。广泛的实验表明,拟议的指标优于所有属性的最新方法。</s>