Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EdgeServe: A Streaming System for Decentralized Model Serving

Ted Shaowang, Sanjay Krishnan

TL;DR

This paper presents EdgeServe, a distributed streaming system that can serve predictions from machine learning models in real time, and evaluates it on three streaming prediction tasks: human activity recognition, autonomous driving, and network intrusion detection.

Abstract

The relevant features for a machine learning task may arrive as one or more continuous streams of data. Serving machine learning models over streams of data creates a number of interesting systems challenges in managing data routing, time-synchronization, and rate control. This paper presents EdgeServe, a distributed streaming system that can serve predictions from machine learning models in real time. We evaluate EdgeServe on three streaming prediction tasks: (1) human activity recognition, (2) autonomous driving, and (3) network intrusion detection.

EdgeServe: A Streaming System for Decentralized Model Serving

TL;DR

This paper presents EdgeServe, a distributed streaming system that can serve predictions from machine learning models in real time, and evaluates it on three streaming prediction tasks: human activity recognition, autonomous driving, and network intrusion detection.

Abstract

The relevant features for a machine learning task may arrive as one or more continuous streams of data. Serving machine learning models over streams of data creates a number of interesting systems challenges in managing data routing, time-synchronization, and rate control. This paper presents EdgeServe, a distributed streaming system that can serve predictions from machine learning models in real time. We evaluate EdgeServe on three streaming prediction tasks: (1) human activity recognition, (2) autonomous driving, and (3) network intrusion detection.
Paper Structure (35 sections, 5 equations, 15 figures, 8 tables)

This paper contains 35 sections, 5 equations, 15 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background and Existing Model Serving Frameworks
  3. What is Streaming Inference?
  4. EdgeServe Architecture and API
  5. EdgeServe Overall Workflow
  6. Execution Layer API
  7. Highly-Responsive Data Stream Joins
  8. Time-triggered Join
  9. Data-triggered Join
  10. When Is Data-triggered Join Better
  11. Lazy Data Routing
  12. Prediction Rate Control
  13. Message Skew
  14. Hybrid Time- and Data-triggered Join
  15. Freshness Threshold
  16. ...and 20 more sections

Figures (15)

  • Figure 1: EdgeServe's execution layer. Data from data streams 1 and 2 (D1, D2) are paired together since both streams are grouped into topic A.
  • Figure 2: Time-triggered join.
  • Figure 3: Data-triggered join.
  • Figure 4: A figure illustrating the order of operations in the lazy data routing system used by EdgeServe.
  • Figure 5: Measure of backlog in the activity recognition task. More frequent predictions are on the left side.
  • ...and 10 more figures

Theorems & Definitions (2)

  • Example 1
  • Example 2