Optimizing Communication for Latency Sensitive HPC Applications on up to 48 FPGAs Using ACCL

TL;DR

This work evaluates ACCL, a packet-switched, MPI-like inter-FPGA communication library, against host MPI approaches for latency-sensitive HPC workloads. By comparing buffered and streaming ACCL configurations and validating with a multi-FPGA shallow water simulation, the authors demonstrate that PL-based scheduling and AXI-stream interfacing dramatically reduce inter-FPGA latency and improve scaling across up to 48 FPGAs on Noctua 2. The study provides latency and throughput models, highlights resource trade-offs (UDP vs TCP, with or without plugins), and shows that ACCL enables near-linear weak scaling and substantial performance gains for latency-bound applications. Overall, ACCL’s configurability and packet-switched network support are shown to be crucial for achieving high performance in large multi-FPGA deployments.

Abstract

Most FPGA boards in the HPC domain are well-suited for parallel scaling because of the direct integration of versatile and high-throughput network ports. However, the utilization of their network capabilities is often challenging and error-prone because the whole network stack and communication patterns have to be implemented and managed on the FPGAs. Also, this approach conceptually involves a trade-off between the performance potential of improved communication and the impact of resource consumption for communication infrastructure, since the utilized resources on the FPGAs could otherwise be used for computations. In this work, we investigate this trade-off, firstly, by using synthetic benchmarks to evaluate the different configuration options of the communication framework ACCL and their impact on communication latency and throughput. Finally, we use our findings to implement a shallow water simulation whose scalability heavily depends on low-latency communication. With a suitable configuration of ACCL, good scaling behavior can be shown to all 48 FPGAs installed in the system. Overall, the results show that the availability of inter-FPGA communication frameworks as well as the configurability of framework and network stack are crucial to achieve the best application performance with low latency communication.

Figures (12)

• Figure 1: Buffered communication controlled by the host. Data is exchanged between ACCL and user kernels via global memory.
• Figure 2: Streaming communication controlled from PL. Data is exchanged between ACCL and user kernels via AXI streams.
• Figure 4: The network infrastructure of the FPGA nodes within the Noctua 2 cluster. The FPGAs are connected to a dedicated Ethernet switch via their QSFP28 ports. The compute nodes communicate via a separate Infiniband network.
• Figure 5: Resource utilization of the network stack and ACCL on the Alveo U280. The ACCL Minimal versions do not contain the compression and arithmetic plugins. The resource utilization of ACCL (green boxes in Figure \ref{['fig-approaches']}) is highlighted by black boxes. All further resources are consumed by the network stack (blue boxes in Figure \ref{['fig-approaches']}).
• Figure 6: Full-duplex communication latencies using ACCL UDP and TCP network stacks and the discussed communication approaches. The latencies for host-side scheduling are modeled for buffered and streaming communication. Communication via the ethernet switch is marked with ES in the legend. All other measurements are done with directly connected FPGAs.