Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and helps us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.
翻译:联邦学习(FL)为大量客户(如个人设备或组织)提供了有希望的方法,以合作学习一个共享的全球模式,使所有客户受益,同时允许用户保持其本地数据。尽管对研究国家语言规划任务中的FL方法感兴趣,文献中缺乏系统比较和分析。在这里,我们介绍了FedNLP, 即一个基准框架,用以评价四种不同任务拟订方法的联邦学习方法:文本分类、序列标记、问答和后续2eq等。我们提出了基于变换语言模式(如个人装置或组织、BERT、BART)和FL方法(如FAvg、FedOPT等)之间的通用界面,目的是根据各种非IID分区战略开发FL方法。我们与FedNLP的广泛实验为 FedNLP方法提供经验性比较,帮助我们更好地了解这一方向的内在挑战。全面分析指出,开发FL任务时,需要先入手,然后进行激动人心的研究。