Shuffle is a key primitive in large-scale data processing applications that has inspired a myriad of implementations. While previous work has produced breakthroughs in shuffle performance, many applications do not benefit in practice because of the difficulty of evolving existing shuffle systems. Shuffle is often tightly integrated into a framework that offers a higher-level abstraction such as SQL. Integrating new shuffle designs into these frameworks requires significant development effort. Furthermore, distributed shuffle is used by many different end use cases, from high-throughput batch processing to low-latency online aggregation. These different use cases have driven the creation of new application frameworks, each of which must rebuild shuffle from scratch. We enable shuffle flexibility by building distributed shuffle as a library. We use distributed futures as an intermediate layer for building distributed shuffle as a library and show how it enables the shuffle control plane to be decoupled from a common high-performance data plane based on Ray. We present Exoshuffle and show that we can: (1) rewrite previous shuffle optimizations as application-level libraries with an order of magnitude less code, (2) build a shuffle-agnostic data plane that provides performance and scalability competitive with specialized shuffle systems, and (3) enable latest applications such as ML training to easily leverage large-scale distributed shuffle.
翻译:在大规模数据处理应用程序中,洗牌是一个关键的原始程序,它激励了许多执行。虽然以前的工作在洗牌性能方面产生了突破,但许多应用实际上并没有受益,因为现有洗牌系统发展起来很困难。洗牌往往被紧密地纳入一个框架,这个框架提供更高层次的抽象,如SQL。将新的洗牌设计纳入这些框架需要大量的开发努力。此外,分散的洗牌被许多不同的终端使用,从高通量批量处理到低延迟在线汇总。这些不同的使用案例驱动了新应用框架的创建,每个新应用框架都必须从零开始重建洗牌。我们通过将分配的洗牌作为图书馆来提供洗牌灵活性。我们把分配的未来作为一个中间层,用来建造分配的洗牌作为图书馆,并展示它如何使洗牌控制平面与基于Ray的通用高性数据平面脱钩。我们介绍了Exsoshofffle,并展示了我们能够:(1) 将先前的洗牌优化改写成应用程序级库库,其具有最高级性能、最高级性、最高级的平面化的平面系统。