Shared Memory-Aware Latency-Sensitive Message Aggregation for Fine-Grained Communication

Message aggregation is often used with a goal to reduce communication cost in HPC applications. The difference in the order of overhead of sending a message and cost of per byte transferred motivates the need for message aggregation, for several irregular fine-grained messaging applications like graph algorithms and parallel discrete event simulation (PDES). While message aggregation is frequently utilized in "MPI-everywhere" model, to coalesce messages between processes mapped to cores, such aggregation across threads in a process, say in MPI+X models or Charm++ SMP (Shared Memory Parallelism) mode, is often avoided. Within-process coalescing is likely to require synchronization across threads and lead to performance issues from contention. However, as a result, SMP-unaware aggregation mechanisms may not fully utilize aggregation opportunities available to applications in SMP mode. Additionally, while the benefit of message aggregation is often analyzed in terms of reducing the overhead, specifically the per message cost, we also analyze different schemes that can aid in reducing the message latency, ie. the time from when a message is sent to the time when it is received. Message latency can affect several applications like PDES with speculative execution where reducing message latency could result in fewer rollbacks. To address these challenges, in our work, we demonstrate the effectiveness of shared memory-aware message aggregation schemes for a range of proxy applications with respect to messaging overhead and latency.