Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs

Bi Xue, Hong Wu, Lei Chen, Chao Yang, Yiming Ma, Fei Ding, Zhen Wang, Liang Wang, Xiaoheng Mao, Ke Huang, Xialu Li, Peng Xia, Rui Jian, Yanli Zhao, Yanzun Huang, Yijie Deng, Harry Tran, Ryan Chang, Min Yu, Eric Dong, Jiazhou Wang, Qianqian Zhang, Keke Zhai, Hongzhang Yin, Pawel Garbacki, Zheng Fang, Yiyi Pan, Min Ni, Yang Liu

TL;DR

SilverTorch presents a unified, model-based GPU-serving system for large-scale recommendations, replacing separate CPU-based ANN and feature-filtering services with tensor-native, in-model operators. It introduces a Bloom Index for GPU feature filtering and a fused INT8 ANN kernel, co-designed to minimize memory and compute while enabling complex retrieval, including OverArch scoring and a Value Model for multi-task retrieval. The approach yields major performance gains (up to 5.6x latency reduction and 23.7x throughput increase) and significant cost savings (up to 13.35x over CPU baselines) while supporting embedding caching for ESR and multi-task retrieval. These results demonstrate the practicality and impact of GPU-based, end-to-end serving for modern, multi-task recommendation models at industry scale.

Abstract

Serving deep learning based recommendation models (DLRM) at scale is challenging. Existing systems rely on CPU-based ANN indexing and filtering services, suffering from non-negligible costs and forgoing joint optimization opportunities. Such inefficiency makes them difficult to support more complex model architectures, such as learned similarities and multi-task retrieval. In this paper, we propose SilverTorch, a model-based system for serving recommendation models on GPUs. SilverTorch unifies model serving by replacing standalone indexing and filtering services with layers of served models. We propose a Bloom index algorithm on GPUs for feature filtering and a tensor-native fused Int8 ANN kernel on GPUs for nearest neighbor search. We further co-design the ANN search index and filtering index to reduce GPU memory utilization and eliminate unnecessary computation. Benefit from SilverTorch's serving paradigm, we introduce a OverArch scoring layer and a Value Model to aggregate results across multi-tasks. These advancements improve the accuracy for retrieval and enable future studies for serving more complex models. For ranking, SilverTorch's design accelerates item embedding calculation by caching the pre-calculated embeddings inside the serving model. Our evaluation on the industry-scale datasets show that SilverTorch achieves up to 5.6x lower latency and 23.7x higher throughput compared to the state-of-the-art approaches. We also demonstrate that SilverTorch's solution is 13.35x more cost-efficient than CPU-based solution while improving accuracy via serving more complex models. SilverTorch serves over hundreds of models online across major products and recommends contents for billions of daily active users.

SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs

TL;DR

SilverTorch presents a unified, model-based GPU-serving system for large-scale recommendations, replacing separate CPU-based ANN and feature-filtering services with tensor-native, in-model operators. It introduces a Bloom Index for GPU feature filtering and a fused INT8 ANN kernel, co-designed to minimize memory and compute while enabling complex retrieval, including OverArch scoring and a Value Model for multi-task retrieval. The approach yields major performance gains (up to 5.6x latency reduction and 23.7x throughput increase) and significant cost savings (up to 13.35x over CPU baselines) while supporting embedding caching for ESR and multi-task retrieval. These results demonstrate the practicality and impact of GPU-based, end-to-end serving for modern, multi-task recommendation models at industry scale.

Abstract

Serving deep learning based recommendation models (DLRM) at scale is challenging. Existing systems rely on CPU-based ANN indexing and filtering services, suffering from non-negligible costs and forgoing joint optimization opportunities. Such inefficiency makes them difficult to support more complex model architectures, such as learned similarities and multi-task retrieval. In this paper, we propose SilverTorch, a model-based system for serving recommendation models on GPUs. SilverTorch unifies model serving by replacing standalone indexing and filtering services with layers of served models. We propose a Bloom index algorithm on GPUs for feature filtering and a tensor-native fused Int8 ANN kernel on GPUs for nearest neighbor search. We further co-design the ANN search index and filtering index to reduce GPU memory utilization and eliminate unnecessary computation. Benefit from SilverTorch's serving paradigm, we introduce a OverArch scoring layer and a Value Model to aggregate results across multi-tasks. These advancements improve the accuracy for retrieval and enable future studies for serving more complex models. For ranking, SilverTorch's design accelerates item embedding calculation by caching the pre-calculated embeddings inside the serving model. Our evaluation on the industry-scale datasets show that SilverTorch achieves up to 5.6x lower latency and 23.7x higher throughput compared to the state-of-the-art approaches. We also demonstrate that SilverTorch's solution is 13.35x more cost-efficient than CPU-based solution while improving accuracy via serving more complex models. SilverTorch serves over hundreds of models online across major products and recommends contents for billions of daily active users.

Paper Structure

This paper contains 28 sections, 4 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Multi-stage Recommendation Flow
  4. Limits of Service-based Recommendation
  5. Limitation of CPU-based Compute
  6. Related Work
  7. SilverTorch Overview
  8. SilverTorch Model Design
  9. Bloom Index
  10. Fused Int8 ANN Search
  11. ANN Search and Filtering Co-design
  12. Extensibility
  13. OverArch Scoring
  14. Multi-Task Retrieval with Value Model
  15. Scale Out
  16. ...and 13 more sections

Figures (11)

  • Figure 1: Motivation Examples: Figure 1(a) shows the service-based recommendation serving. Figure 1(b) shows the versioning challenge of service-based retrieval.
  • Figure 2: Overall workflow of SilverTorch model publish and serving and Pseudocode of SilverTorch model definition.
  • Figure 3: Bloom Index Design for Feature Filtering
  • Figure 4: An example for co-designed ANN with Filtering.
  • Figure 5: End-to-end performance Results for Retrieval and ESR.
  • ...and 6 more figures