We propose a federated averaging Langevin algorithm (FA-LD) for uncertainty quantification and mean predictions with distributed clients. In particular, we generalize beyond normal posterior distributions and consider a general class of models. We develop theoretical guarantees for FA-LD for strongly log-concave distributions with non-i.i.d data and study how the injected noise and the stochastic-gradient noise, the heterogeneity of data, and the varying learning rates affect the convergence. Such an analysis sheds light on the optimal choice of local updates to minimize communication costs. Important to our approach is that the communication efficiency does not deteriorate with the injected noise in the Langevin algorithms. In addition, we examine in our FA-LD algorithm both independent and correlated noise used over different clients. We observe that there is also a trade-off between federation and communication cost there. As local devices may become inactive in the federated network, we also show convergence results based on different averaging schemes where only partial device updates are available.
翻译:我们建议采用联盟平均朗埃文算法(FA-LD)来对不确定性进行量化,并与分布式客户进行平均预测。特别是,我们普遍采用超出正常的后方分布,并考虑一般的模型类别。我们为FA-LD制定了理论保障,以便与非i.i.d.数据和研究注射噪音和随机偏移噪音、数据的异质性以及不同的学习率如何影响趋同。这种分析为当地更新的最佳选择提供了线索,以尽量减少通信成本。我们的方法很重要的是,Langevin算法中注入的噪音不会使通信效率恶化。此外,我们在我们的FA-LD算法中检查了不同客户所使用的独立和关联的噪音。我们发现,那里的联邦和通信成本之间也有某种权衡。由于联邦网络中的当地设备可能变得不活跃,我们还显示了基于不同平均方法的趋同结果,因为只有局部的设备更新。