Large scale contextual representation models have significantly advanced NLP in recent years, understanding the semantics of text to a degree never seen before. However, they need to process large amounts of data to achieve high-quality results. Joining and accessing all these data from multiple sources can be extremely challenging due to privacy and regulatory reasons. Federated Learning can solve these limitations by training models in a distributed fashion, taking advantage of the hardware of the devices that generate the data. We show the viability of training NLP models, specifically Word2Vec, with the Federated Learning protocol. In particular, we focus on a scenario in which a small number of organizations each hold a relatively large corpus. The results show that neither the quality of the results nor the convergence time in Federated Word2Vec deteriorates as compared to centralised Word2Vec.
翻译:近年来,大型背景代表模式大大推进了国家学习计划,理解了文本的语义,达到了前所未有的程度,然而,它们需要处理大量数据,以取得高质量的成果。由于隐私和监管原因,从多种来源合并和获取所有这些数据可能极具挑战性。联邦学习联盟可以利用生成数据的设备的硬件,以分布方式通过培训模式解决这些局限性。我们展示了培训国家学习计划模式的可行性,特别是Word2Vec,与联邦学习协议相结合。特别是,我们侧重于少数组织各自拥有相对大体的情景。结果显示,与中央化的Word2Vec相比,联邦Word2Vec系统的结果质量和趋同时间都没有恶化。