Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skyrise: Exploiting Serverless Cloud Infrastructure for Elastic Data Processing

Thomas Bodner, Daniel Ritter, Martin Boissier, Tilmann Rabl

TL;DR

Skyrise tackles the lack of end-to-end serverless SQL processing by building the first fully serverless SQL engine on top of FaaS compute and serverless storage. It employs adaptive and cost-aware techniques, including retriggering of stragglers, a cache of intermediate results in serverless storage, and tiered shuffles to cope with performance variability and storage bottlenecks. Evaluations on the TPC-H benchmark show Skyrise achieving competitive latency and cost relative to both academic prototypes and commercial cloud systems for terabyte-scale queries, while maintaining elasticity from zero to large workloads. This work demonstrates the practicality of serverless SQL for analytics and provides an open-source platform to drive further research and adoption.

Abstract

Serverless computing offers elasticity unmatched by conventional server-based cloud infrastructure. Although modern data processing systems embrace serverless storage, such as Amazon S3, they continue to manage their compute resources as servers. This is challenging for unpredictable workloads, leaving clusters often underutilized. Recent research shows the potential of serverless compute resources, such as cloud functions, for elastic data processing, but also sees limitations in performance robustness and cost efficiency for long running workloads. These challenges require holistic approaches across the system stack. However, to the best of our knowledge, there is no end-to-end data processing system built entirely on serverless infrastructure. In this paper, we present Skyrise, our effort towards building the first fully serverless SQL query processor. Skyrise exploits the elasticity of its underlying infrastructure, while alleviating the inherent limitations with a number of adaptive and cost-aware techniques. We show that both Skyrise's performance and cost are competitive to other cloud data systems for terabyte-scale queries of the analytical TPC-H benchmark.

Skyrise: Exploiting Serverless Cloud Infrastructure for Elastic Data Processing

TL;DR

Skyrise tackles the lack of end-to-end serverless SQL processing by building the first fully serverless SQL engine on top of FaaS compute and serverless storage. It employs adaptive and cost-aware techniques, including retriggering of stragglers, a cache of intermediate results in serverless storage, and tiered shuffles to cope with performance variability and storage bottlenecks. Evaluations on the TPC-H benchmark show Skyrise achieving competitive latency and cost relative to both academic prototypes and commercial cloud systems for terabyte-scale queries, while maintaining elasticity from zero to large workloads. This work demonstrates the practicality of serverless SQL for analytics and provides an open-source platform to drive further research and adoption.

Abstract

Serverless computing offers elasticity unmatched by conventional server-based cloud infrastructure. Although modern data processing systems embrace serverless storage, such as Amazon S3, they continue to manage their compute resources as servers. This is challenging for unpredictable workloads, leaving clusters often underutilized. Recent research shows the potential of serverless compute resources, such as cloud functions, for elastic data processing, but also sees limitations in performance robustness and cost efficiency for long running workloads. These challenges require holistic approaches across the system stack. However, to the best of our knowledge, there is no end-to-end data processing system built entirely on serverless infrastructure. In this paper, we present Skyrise, our effort towards building the first fully serverless SQL query processor. Skyrise exploits the elasticity of its underlying infrastructure, while alleviating the inherent limitations with a number of adaptive and cost-aware techniques. We show that both Skyrise's performance and cost are competitive to other cloud data systems for terabyte-scale queries of the analytical TPC-H benchmark.
Paper Structure (19 sections, 7 figures, 3 tables)

This paper contains 19 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Serverless Infrastructure
  3. Function as a Service Platforms
  4. Serverless Storage Services
  5. System Architecture
  6. General Overview
  7. Query Compilation
  8. Query Execution
  9. Storage Management
  10. Experimental Evaluation
  11. Setup and Methodology
  12. End-to-End Workloads
  13. Query Latency
  14. Monetary Cost
  15. Elasticity
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Architecture of the Skyrise query processor showing the component interactions for processing a query.
  • Figure 2: Query compilation pipeline showing the intermediate and final compilation artifacts and their consumption.
  • Figure 3: Query execution plan for TPC-H Q12.
  • Figure 4: Storage stack of the Skyrise query worker including the interactions between query operators and cloud storage.
  • Figure 5: Query performance for TPC-H queries.
  • ...and 2 more figures