Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs

Cloud datacenters are growing exponentially both in number and size. As communication protocols evolve, datacenter networks experience higher utilization, leading to greater congestion along with increased latency and packet loss. We analyze a recently published reinforcement learning congestion control algorithm (Tessler et al. 2022) that achieves state-of-the-art performance and, in a second phase, reshape it to comply with current hardware limitations. We show how to map complex policies to a low-compute architecture, gaining x500 latency reduction. This transformation enables real-time policy inference within the $\mu$sec decision time requirement, with a negligible effect on the quality of the policy. We deploy the transformed policy onto NVIDIA NICs in an operational network. Compared to popular CC algorithms used in production, we show that RL-CC is the only one to perform well on all benchmarks tested, balancing multiple metrics simultaneously: bandwidth, latency, and packet drops. This sheds light on the feasibility of data-driven methods for congestion control, challenging the prior belief that handcrafted heuristics are required to obtain a stable and fair solution.