Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication

Joe Oakley, Hakan Ferhatosmanoglu

TL;DR

This work tackles the challenge of scaling ML inference on serverless platforms by introducing FSD-Inference, the first fully serverless, MPI-like distributed inference system. It combines two novel cloud IPC channels—FSD-Inf-Queue (pub-sub/queueing) and FSD-Inf-Object (object storage)—with a hierarchical FaaS launch and sparsity-aware, layer-wise model partitioning to enable intra-layer parallelism. A formal end-to-end cost model, along with optimizations like payload packing, long polling, and hypergraph partitioning, demonstrates favorable cost-to-performance across varying model sizes and workloads, including sporadic bursts. Empirical results against serverful baselines and HPC show that FSD-Inference can achieve competitive latency and dramatically improved cost efficiency, validating the practicality of fully serverless distributed ML inference for large DNNs and LLM-scale models in real-world cloud environments.

Abstract

Serverless computing offers attractive scalability, elasticity and cost-effectiveness. However, constraints on memory, CPU and function runtime have hindered its adoption for data-intensive applications and machine learning (ML) workloads. Traditional 'server-ful' platforms enable distributed computation via fast networks and well-established inter-process communication (IPC) mechanisms such as MPI and shared memory. In the absence of such solutions in the serverless domain, parallel computation with significant IPC requirements is challenging. We present FSD-Inference, the first fully serverless and highly scalable system for distributed ML inference. We explore potential communication channels, in conjunction with Function-as-a-Service (FaaS) compute, to design a state-of-the-art solution for distributed ML within the context of serverless data-intensive computing. We introduce novel fully serverless communication schemes for ML inference workloads, leveraging both cloud-based publish-subscribe/queueing and object storage offerings. We demonstrate how publish-subscribe/queueing services can be adapted for FaaS IPC with comparable performance to object storage, while offering significantly reduced cost at high parallelism levels. We conduct in-depth experiments on benchmark DNNs of various sizes. The results show that when compared to server-based alternatives, FSD-Inference is significantly more cost-effective and scalable, and can even achieve competitive performance against optimized HPC solutions. Experiments also confirm that our serverless solution can handle large distributed workloads and leverage high degrees of FaaS parallelism.

FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication

TL;DR

This work tackles the challenge of scaling ML inference on serverless platforms by introducing FSD-Inference, the first fully serverless, MPI-like distributed inference system. It combines two novel cloud IPC channels—FSD-Inf-Queue (pub-sub/queueing) and FSD-Inf-Object (object storage)—with a hierarchical FaaS launch and sparsity-aware, layer-wise model partitioning to enable intra-layer parallelism. A formal end-to-end cost model, along with optimizations like payload packing, long polling, and hypergraph partitioning, demonstrates favorable cost-to-performance across varying model sizes and workloads, including sporadic bursts. Empirical results against serverful baselines and HPC show that FSD-Inference can achieve competitive latency and dramatically improved cost efficiency, validating the practicality of fully serverless distributed ML inference for large DNNs and LLM-scale models in real-world cloud environments.

Abstract

Serverless computing offers attractive scalability, elasticity and cost-effectiveness. However, constraints on memory, CPU and function runtime have hindered its adoption for data-intensive applications and machine learning (ML) workloads. Traditional 'server-ful' platforms enable distributed computation via fast networks and well-established inter-process communication (IPC) mechanisms such as MPI and shared memory. In the absence of such solutions in the serverless domain, parallel computation with significant IPC requirements is challenging. We present FSD-Inference, the first fully serverless and highly scalable system for distributed ML inference. We explore potential communication channels, in conjunction with Function-as-a-Service (FaaS) compute, to design a state-of-the-art solution for distributed ML within the context of serverless data-intensive computing. We introduce novel fully serverless communication schemes for ML inference workloads, leveraging both cloud-based publish-subscribe/queueing and object storage offerings. We demonstrate how publish-subscribe/queueing services can be adapted for FaaS IPC with comparable performance to object storage, while offering significantly reduced cost at high parallelism levels. We conduct in-depth experiments on benchmark DNNs of various sizes. The results show that when compared to server-based alternatives, FSD-Inference is significantly more cost-effective and scalable, and can even achieve competitive performance against optimized HPC solutions. Experiments also confirm that our serverless solution can handle large distributed workloads and leverage high degrees of FaaS parallelism.
Paper Structure (48 sections, 4 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 48 sections, 4 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Designing a Cloud-based Serverless ML Solution
  3. Serverless Compute
  4. Cloud-based Distributed ML
  5. Model Partitioning in FaaS Settings
  6. Serverless Communication
  7. Data Streaming Systems
  8. ETL on Streaming Data
  9. NoSQL Databases
  10. Publish-Subscribe
  11. Message Queues
  12. Object Storage
  13. FSD-Inference Design
  14. FSD-Inf-Queue Communication Channel
  15. FSD-Inf-Object Communication Channel
  16. ...and 33 more sections

Figures (6)

  • Figure 1: FSD-Inference high-level cloud architecture
  • Figure 2: FSD-Inf-Queue communication scheme. Demonstrated with AWS services (SNS, SQS, Lambda).
  • Figure 3: FSD-Inf-Object communication scheme. Demonstrated with AWS services.
  • Figure 4: Daily cost of FSD-Inference, Server-Always-On and Server-Job-Scoped for various query volumes. Queries are evenly spread between $N=1024, 4096, 16384$ and $65536$.
  • Figure 5: Query latency for FSD-Inference, server-based baselines and H-SpFF.
  • ...and 1 more figures