Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs

Model hubs with many pre-trained models (PTMs) have been a cornerstone in deep learning. Although built at a high cost, they remain \emph{under-exploited}: practitioners usually pick one PTM from the provided model hub by popularity and then fine-tune the PTM to solve the target task. This na\"ive but common practice poses two obstacles to sufficient exploitation of pre-trained model hubs: (1) the PTM selection by popularity has no optimality guarantee; (2) only one PTM is used while the rest PTMs are ignored. Ideally, to exploit pre-trained model hubs maximally, trying all combinations of PTMs and extensively fine-tuning each PTM combination are required, which incurs exponential combinations and an unaffordable computational budget. In this paper, we propose a new paradigm of exploiting model hubs by ranking and tuning pre-trained models: (1) Our conference paper~\citep{you_logme:_2021} proposed LogME to estimate the maximum value of label evidence given features extracted by pre-trained models, which can rank all the PTMs in a model hub for various types of PTMs and tasks \emph{before fine-tuning}. (2) The best ranked PTM can be fine-tuned and deployed if we have no preference for the model's architecture, or the target PTM can be tuned by top-K ranked PTMs via the proposed B-Tuning algorithm. The ranking part is based on the conference paper, and we complete its theoretical analyses in this paper, including the convergence proof of the heuristic evidence maximization procedure and the influence of feature dimension. The tuning part introduces a novel Bayesian Tuning (B-Tuning) method for tuning multiple PTMs, which surpasses specialized methods designed for tuning homogeneous PTMs and sets up a new state of the art for tuning heterogeneous PTMs. The new paradigm of exploiting PTM hubs can be interesting to a large audience across the machine learning community.