Large pretrained language models have been widely used in downstream NLP tasks via task-specific fine-tuning. Recently, an array of Parameter-Efficient Fine-Tuning (PEFT) methods have also achieved strong task performance while updating a much smaller number of parameters compared to full model tuning. However, it is non-trivial to make informed per-task design choices (i.e., to create PEFT configurations) concerning the selection of PEFT architectures and modules, the number of tunable parameters, and even the layers in which the PEFT modules are inserted. Consequently, it is highly likely that the current, manually set PEFT configurations might be suboptimal for many tasks from the perspective of the performance-to-efficiency trade-off. To address the core question of the PEFT configuration selection that aims to control and maximise the balance between performance and parameter efficiency, we first define a rich configuration search space spanning multiple representative PEFT modules along with finer-grained configuration decisions over the modules (e.g., parameter budget, insertion layer). We then propose AutoPEFT, a novel framework to traverse this configuration space: it automatically configures multiple PEFT modules via high-dimensional Bayesian optimisation. We show the resource scalability and task transferability of AutoPEFT-found configurations, outperforming existing PEFT methods on average on the standard GLUE benchmark while conducting the configuration search on a single task. The per-task AutoPEFT-based configuration search even outperforms full-model fine-tuning.
翻译:在下游的NLP任务中,通过针对具体任务的微调,广泛使用了大型预先培训的语言模型。最近,一系列的Parameter-Efficient Fine-Tuning(PEFT)方法也取得了很强的任务性能,同时从业绩到效率全面调整的角度更新了大量参数。然而,在选择PEFT架构和模块、金枪鱼参数数量,甚至插入PEFT模块的层次方面,作出知情的每个任务设计选择(即创建PEFT配置)是非三边性的。因此,从业绩到效率全面调整的角度,目前手工设置的PEEFT配置也很可能不理想。然而,为了解决PEEFT配置选择的核心问题,目的是控制并最大限度地实现业绩和参数效率之间的平衡,我们首先确定一个内容丰富的配置搜索空间,覆盖多个具有代表性的PEFT模块,以及基于精细的配置决定(例如,参数预算,插入层),因此,现在很可能是手动式的PEFTFT配置。我们随后提议一个通过高级的系统平均配置格式,然后通过S-FTFFT任务向当前的自动配置结构转换一个新的框架框架框架,我们提出一个通过Slovelopferfterforferal-forforti 。