Multi-Tenant SmartNICs for In-Network Preprocessing of Recommender Systems

TL;DR

The paper tackles the preprocessing bottleneck in large-scale recommender systems by introducing Piper, a network-attached FPGA accelerator that streams data loading and preprocessing in a fully pipelined fashion. It combines in-memory and in-network memory access with dynamic MiniPipe units to run multiple pipelines concurrently, delivering substantial speedups over CPU and GPU baselines while markedly improving energy efficiency. Key innovations include dense and sparse feature operators, vocabulary-table based embedding preprocessing, and runtime reconfigurability through partial reconfiguration. The results show Piper achieving up to $39\sim105\times$ CPU speedups and $3\sim17\times$ GPU speedups, with significant reductions in power consumption, demonstrating a practical, scalable path for hybrid cloud data centers to handle dynamic data and evolving preprocessing workloads.

Abstract

Keeping ML-based recommender models up-to-date as data drifts and evolves is essential to maintain accuracy. As a result, online data preprocessing plays an increasingly important role in serving recommender systems. Existing solutions employ multiple CPU workers to saturate the input bandwidth of a single training node. Such an approach results in high deployment costs and energy consumption. For instance, a recent report from industrial deployments shows that data storage and ingestion pipelines can account for over 60\% of the power consumption in a recommender system. In this paper, we tackle the issue from a hardware perspective by introducing Piper, a flexible and network-attached accelerator that executes data loading and preprocessing pipelines in a streaming fashion. As part of the design, we define MiniPipe, the smallest pipeline unit enabling multi-pipeline implementation by executing various data preprocessing tasks across the single board, giving Piper the ability to be reconfigured at runtime. Our results, using publicly released commercial pipelines, show that Piper, prototyped on a power-efficient FPGA, achieves a 39$\sim$105$\times$ speedup over a server-grade, 128-core CPU and 3$\sim$17$\times$ speedup over GPUs like RTX 3090 and A100 in multiple pipelines. The experimental analysis demonstrates that Piper provides advantages in both latency and energy efficiency for preprocessing tasks in recommender systems, providing an alternative design point for systems that today are in very high demand.

Figures (12)

• Figure 1: Performance analysis for DLRM training in A100 machine, including preprocessing phase (CPU) and training phase (CPU for data loading, GPU for training).
• Figure 2: System overview for distributed online training of ML-based recommender models. Engineers regularly collect data from various sources, store it in distributed file systems, process it through distributed preprocessing nodes, and finally send it to computing servers to update the pre-trained models.
• Figure 3: Various configurations for distributed ML training systems where Piper can work in both in-memory or in-network modes.
• Figure 4: Data access pattern for PCIe-attached hardware accelerator in the host system.
• Figure 5: Hierarchical abstractions for memory access.