Application-Centric Benchmarking of Distributed FaaS Platforms using BeFaaS

TL;DR

BeFaaS introduces an application-centric benchmarking framework for distributed FaaS platforms, extending prior work with federated cloud/edge deployments, asynchronous event pipelines, and four realistic benchmarks. The framework automates deployment, execution, and fine-grained data collection, enabling drill-down analysis of function chains, cold starts, and cross-provider interactions. Through four experiments across AWS, Azure, GCP, and tinyFaaS, the study reveals that network latency and inter-provider event delays often dominate performance, with edge deployments reducing total latency in certain setups and provider-specific cold-start behavior varying notably. BeFaaS offers a practical, extensible tool for developers to compare offerings and for researchers to study deployment effects across cloud, edge, and fog environments; it is open-source and adaptable to new workloads and platforms.

Abstract

Due to the popularity of the FaaS programming model, there is now a wide variety of commercial and open-source FaaS systems. Hence, for comparison of different FaaS systems and their configuration options, FaaS application developers rely on FaaS benchmarking frameworks. Existing frameworks, however, tend to evaluate only single isolated aspects, a more holistic application-centric benchmarking framework is still missing. In previous work, we proposed BeFaaS, an extensible application-centric benchmarking framework for FaaS environments that focuses on the evaluation of FaaS platforms through realistic and typical examples of FaaS applications. In this extended paper, we (i) enhance our benchmarking framework with additional features for distributed FaaS setups, (ii) design application benchmarks reflecting typical FaaS use cases, and (iii) use them to run extensive experiments with commercial cloud FaaS platforms (AWS Lambda, Azure Functions, Google Cloud Functions) and the tinyFaaS edge serverless platform. BeFaaS now includes four FaaS application-centric benchmarks, is extensible for additional workload profiles and platforms, and supports federated benchmark runs in which the benchmark application is distributed over multiple FaaS systems while collecting fine-grained measurement results for drill-down analysis. Our experiment results show that (i) network transmission is a major contributor to response latency for function chains, (ii) this effect is exacerbated in hybrid edge-cloud deployments, (iii) the trigger delay between a published event and the start of the triggered function ranges from about 100ms for AWS Lambda to 800ms for Google Cloud Functions, and (iv) Azure Functions shows the best cold start behavior for our workloads.

Figures (17)

• Figure 1: High-level overview of the BeFaaS architecture.
• Figure 2: The Deployment Compiler transforms application code into individual deployment artifacts based on a deployment configuration. These are then deployed and invoked by the Load Generator to retrieve measurement results. Finally, the Benchmark Manager aggregates and reports fine-grained results.
• Figure 3: A publisher function forwards incoming events to the respective event pipeline to trigger the called function.
• Figure 4: The e-commerce application implements a web shop in 17 functions. The Frontend serves as a single entry point and an external database is used to store state. We group some functions to increase legibility.
• Figure 5: The IoT application implements a smart traffic light scenario in nine functions. The Load Generator emulates sensor data and sends them to three different entry points.