In this paper, we consider a hierarchical distributed multi-task learning (MTL) system where distributed users wish to jointly learn different models orchestrated by a central server with the help of a layer of multiple relays. Since the users need to download different learning models in the downlink transmission, the distributed MTL suffers more severely from the communication bottleneck compared to the single-task learning system. To address this issue, we propose a coded hierarchical MTL scheme that exploits the connection topology and introduces coding techniques to reduce communication loads. It is shown that the proposed scheme can significantly reduce the communication loads both in the uplink and downlink transmissions between relays and the server. Moreover, we provide information-theoretic lower bounds on the optimal uplink and downlink communication loads, and prove that the gaps between achievable upper bounds and lower bounds are within the minimum number of connected users among all relays. In particular, when the network connection topology can be delicately designed, the proposed scheme can achieve the information-theoretic optimal communication loads. Experiments on real datasets show that our proposed scheme can reduce the overall training time by 17% $\sim$ 26% compared to the conventional uncoded scheme.
翻译:在本文中,我们考虑一个分级分布式多任务学习(MTL)系统,在这个系统中,分布式用户希望共同学习由中央服务器在多式继电器的帮助下操纵的不同模式。由于用户需要下载下行传输中不同的学习模式,分布式的MTL比单一任务学习系统更严重地受到通信瓶颈的影响。为了解决这个问题,我们建议了一个编码式的分级MTL系统,利用连接表层学和引入编码技术来减少通信负荷。它表明,拟议的计划可以大大减少中继器和服务器之间上行和下行链路传输中的通信负荷。此外,由于用户需要下载下行传输中的不同学习模式,因此分布式的MTL比单一任务学习系统更容易受到通信瓶颈的影响。为了解决这个问题,我们建议了一个编码式的分级MTL系统,利用连接表层学和引入编码技术来减少通信负荷。拟议的计划可以大大减少上行和下行传输中继器之间的通信负荷。此外,我们提供的关于最佳上行和下行通信负荷的信息-理论下行的下行低边框,我们提出的计划可以减少整个上行线路和下线通信负荷的公式的公式,用17\美元来比较常规的公式。