The BRAM is the Limit: Shattering Myths, Shaping Standards, and Building Scalable PIM Accelerators

Many recent FPGA-based Processor-in-Memory (PIM) architectures have appeared with promises of impressive levels of parallelism but with performance that falls short of expectations due to reduced maximum clock frequencies, an inability to scale processing elements up to the maximum BRAM capacity, and minimal hardware support for large reduction operations. In this paper, we first establish what we believe should be a "Gold Standard" set of design objectives for PIM-based FPGA designs. This Gold Standard was established to serve as an absolute metric for comparing PIMs developed on different technology nodes and vendor families as well as an aspirational goal for designers. We then present IMAGine, an In-Memory Accelerated GEMV engine used as a case study to show the Gold Standard can be realized in practice. IMAGine serves as an existence proof that dispels several myths surrounding what is normally accepted as clocking and scaling FPGA performance limitations. Specifically, IMAGine clocks at the maximum frequency of the BRAM and scales to 100% of the available BRAMs. Comparative analyses are presented showing execution speeds over existing PIM-based GEMV engines on FPGAs and achieving a 2.65x - 3.2x faster clock. An AMD Alveo U55 implementation is presented that achieves a system clock speed of 737 MHz, providing 64K bit-serial multiply-accumulate (MAC) units for GEMV operation. This establishes IMAGine as the fastest PIM-based GEMV overlay, outperforming even the custom PIM-based FPGA accelerators reported to date. Additionally, it surpasses TPU v1-v2 and Alibaba Hanguang 800 in clock speed while offering an equal or greater number of MAC units.