SeBS-Flow: Benchmarking Serverless Cloud Function Workflows

TL;DR

The paper addresses the lack of fair, cross-platform benchmarking for serverless workflows by introducing SeBS-Flow, a platform-agnostic model grounded on Petri nets with data (WFD-nets). It defines a portable JSON workflow definition, platform-specific transcription to AWS, Google Cloud, and Azure, and a benchmark suite of six real-world plus four microbenchmarks. Through extensive experiments, it reveals runtime differences, orchestration overhead sources (storage I/O, scheduling, return payload), and the impact of OS noise and cold starts, alongside pricing and time-evolution insights. The work enables reproducible, comparable evaluation of serverless workflows across clouds, aiding developers and researchers in platform selection and performance optimization, and is released as open-source for ongoing benchmarking over time.

Abstract

Serverless computing has emerged as a prominent paradigm, with a significant adoption rate among cloud customers. While this model offers advantages such as abstraction from the deployment and resource scheduling, it also poses limitations in handling complex use cases due to the restricted nature of individual functions. Serverless workflows address this limitation by orchestrating multiple functions into a cohesive application. However, existing serverless workflow platforms exhibit significant differences in their programming models and infrastructure, making fair and consistent performance evaluations difficult in practice. To address this gap, we propose the first serverless workflow benchmarking suite SeBS-Flow, providing a platform-agnostic workflow model that enables consistent benchmarking across various platforms. SeBS-Flow includes six real-world application benchmarks and four microbenchmarks representing different computational patterns. We conduct comprehensive evaluations on three major cloud platforms, assessing performance, cost, scalability, and runtime deviations. We make our benchmark suite open-source, enabling rigorous and comparable evaluations of serverless workflows over time.

Figures (17)

• Figure 1: Workflow invoking function process in parallel, with inputs from zero to three and results written to res.
• Figure 2: WFD-net with transitions $T = \{ t_1, t_2, t_3, t_4 \}$ and places $P = \{ p_1, p_2, p_3, p_4, start, end \}$
• Figure 3: Workflow using our model based on WFD-nets.
• Figure 4: Workflow definition language: a portable specification of control-flow and data dependencies.
• Figure 5: Process of executing a workflow using SeBS-Flow.