Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shared Memory-Aware Latency-Sensitive Message Aggregation for Fine-Grained Communication

Kavitha Chandrasekar, Laxmikant Kale

TL;DR

This work demonstrates the effectiveness of process-aware message aggregation schemes for a range of proxy applications with respect to messaging overhead and latency.

Abstract

Message aggregation is often used with a goal to reduce communication cost in HPC applications. The difference in the order of overhead of sending a message and cost of per byte transferred motivates the need for message aggregation, for several irregular fine-grained messaging applications like graph algorithms and parallel discrete event simulation (PDES). While message aggregation is frequently utilized in "MPI-everywhere" model, to coalesce messages between processes mapped to cores, such aggregation across threads in a process, say in MPI+X models or Charm++ SMP (Shared Memory Parallelism) mode, is often avoided. Within-process coalescing is likely to require synchronization across threads and lead to performance issues from contention. However, as a result, SMP-unaware aggregation mechanisms may not fully utilize aggregation opportunities available to applications in SMP mode. Additionally, while the benefit of message aggregation is often analyzed in terms of reducing the overhead, specifically the per message cost, we also analyze different schemes that can aid in reducing the message latency, ie. the time from when a message is sent to the time when it is received. Message latency can affect several applications like PDES with speculative execution where reducing message latency could result in fewer rollbacks. To address these challenges, in our work, we demonstrate the effectiveness of shared memory-aware message aggregation schemes for a range of proxy applications with respect to messaging overhead and latency.

Shared Memory-Aware Latency-Sensitive Message Aggregation for Fine-Grained Communication

TL;DR

This work demonstrates the effectiveness of process-aware message aggregation schemes for a range of proxy applications with respect to messaging overhead and latency.

Abstract

Message aggregation is often used with a goal to reduce communication cost in HPC applications. The difference in the order of overhead of sending a message and cost of per byte transferred motivates the need for message aggregation, for several irregular fine-grained messaging applications like graph algorithms and parallel discrete event simulation (PDES). While message aggregation is frequently utilized in "MPI-everywhere" model, to coalesce messages between processes mapped to cores, such aggregation across threads in a process, say in MPI+X models or Charm++ SMP (Shared Memory Parallelism) mode, is often avoided. Within-process coalescing is likely to require synchronization across threads and lead to performance issues from contention. However, as a result, SMP-unaware aggregation mechanisms may not fully utilize aggregation opportunities available to applications in SMP mode. Additionally, while the benefit of message aggregation is often analyzed in terms of reducing the overhead, specifically the per message cost, we also analyze different schemes that can aid in reducing the message latency, ie. the time from when a message is sent to the time when it is received. Message latency can affect several applications like PDES with speculative execution where reducing message latency could result in fewer rollbacks. To address these challenges, in our work, we demonstrate the effectiveness of shared memory-aware message aggregation schemes for a range of proxy applications with respect to messaging overhead and latency.

Paper Structure

This paper contains 11 sections, 18 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Aggregation Schemes
  4. SMP woes for fine-grained computations
  5. Basic Aggregation Schemes for aggregation in SMP context
  6. Analysis
  7. Benchmarks and Applications
  8. Evaluation
  9. System and benchmarks
  10. Related Work
  11. Conclusion and Future Work

Figures (18)

  • Figure 1: Time to send a message does not change for small byte count, suggesting time is dominated by latency $\alpha$ of $\mu$seconds
  • Figure 2: PingAck benchmark
  • Figure 3: PingAck benchmark SMP (different process counts) vs non-SMP on 2 physical nodes
  • Figure 4: WW: Source worker maintains a buffer per destination worker
  • Figure 5: WPs: Source worker maintains a buffer per destination $process$
  • ...and 13 more figures