With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.
翻译:近年来,随着经过预先培训的模型越来越多,为某一特定下游分类任务选择经过培训的最佳检查站的问题日益引起人们的注意,虽然最近提出了几种方法(例如LEEP、H-score)以解决选择问题,但这些方法采用学习理论没有很好动力的超常学方法。在本文中,我们介绍了PACTran,这是一套具有理论基础的用于事先培训的模式选择和可转移性衡量的衡量标准。我们首先展示了如何从转移学习环境下的最佳PAC-Bayesian约束中得出PACTran指标。我们随后对PACTran关于一些愿景任务(VTAB)以及语言和视觉任务(OKVQA)的三种衡量速率进行了经验性评估。对结果的分析表明,PACTran与现有的选择方法相比,一个更加一致和有效的可转移性措施。