Large ML models and datasets have necessitated the use of multi-GPU systems for distributed model training. To harness the power offered by multi-GPU systems, it is critical to eliminate bottlenecks in inter-GPU communication - a problem made challenging by the heterogeneous nature of interconnects. In this work, we present TACCL, a synthesizer for collective communication primitives for large-scale multi-GPU systems. TACCL encodes a profiled topology and input size into a synthesis problem to generate optimized communication algorithms. TACCL is built on top of the standard NVIDIA Collective Communication Library (NCCL), allowing it to be a drop-in replacement for GPU communication in frameworks like PyTorch with minimal changes. TACCL generates algorithms for communication primitives like Allgather, Alltoall, and Allreduce that are up to $3\times$ faster than NCCL. Using TACCL's algorithms speeds up the end-to-end training of an internal mixture of experts model by $17\%$. By decomposing the optimization problem into parts and leveraging the symmetry in multi-GPU topologies, TACCL synthesizes collectives for up to 80-GPUs in less than 3 minutes, at least two orders of magnitude faster than other synthesis-based state-of-the-art collective communication libraries.
翻译:大型 ML 模型和数据集使得有必要使用多GPU系统进行分布式模型培训。 要利用多GPU系统提供的力量, 关键是要消除GPU之间通信的瓶颈 -- -- 这是因互连性质差异性造成的问题。 在这项工作中, 我们为大型多GPU系统的集体通信原始功能提供TACCL 合成器。 TACCL 将剖析表层和输入大小编码为合成问题, 以产生优化的通信算法。 TACCL 在标准 NVIDIA 集体通信图书馆( NCCL) 的顶端上建了 TACCL, 使其能够在像 PyTorrch 这样的框架中, 成为 GPUPL 通信的缩放替换器。 TACCL 为 Allgather, Alltoall 和 Alled 等通信原始元素生成算法, 比NCLCL 更快3 。 使用 TACCL 的算法加快了对内部专家混合模型的端到端培训, 17 ⁇ $。 将优化问题解到最小部分, 将GPIPL 3 快速的合成库中, 以比 80 最高合成速度更快的合成库中 。