Lightning: Scaling the GPU Programming Model Beyond a Single GPU

The GPU programming model is primarily designed to support the development of applications that run on one GPU. However, just a single GPU is limited in its capabilities in terms of memory capacity and compute power. To handle large problems that exceed these capabilities, one must rewrite application code to manually transfer data between GPU memory and higher-level memory and/or distribute the work across multiple GPUs, possibly in multiple nodes. This means a large engineering effort is required to scale GPU applications beyond a single GPU. We present Lightning: a framework that follows the common GPU programming paradigm, but enables scaling to larger problems. Lightning enables multi-GPU execution of GPU kernels, even across multiple nodes, and seamlessly spills data to main memory and disk when required. Existing CUDA kernels can easily be adapted for use in Lightning, with data access annotations on these kernels allowing Lightning to infer their data requirements and dependencies. Lightning efficiently distributes the work/data across GPUs and maximizes efficiency by overlapping scheduling, data movement, and work when possible. We present the design and implementation of Lightning, as well as experimental results on up to 32 GPUs for eight benchmarks and an application from geospatial clustering. Evaluation shows excellent performance on problem sizes that far exceed the memory capacity of a single GPU.