Unveiling Overlooked Performance Variance in Serverless Computing
Jinfeng Wen, Zhenpeng Chen, Federica Sarro, Shangguang Wang
TL;DR
This paper addresses the overlooked problem of performance variance in serverless computing. By combining a literature survey of 99 papers with an empirical measurement of 72 serverless functions across multiple runs, it reveals large end-to-end latency variance (up to 338.76% with an average around 44%), and shows that most studies use too few repetitions, jeopardizing reproducibility. The work provides a detailed characterization of variance (via CV and boxplots), demonstrates that warm starts often exhibit greater variability, and highlights that reliable performance requires substantially more repetitions than commonly used in the literature. The findings carry practical implications for researchers, developers, and cloud providers, urging variance-aware benchmarking and offering a replication package to foster reproducible, reliable serverless performance evaluation.
Abstract
Serverless computing is an emerging cloud computing paradigm for developing applications at the function level, known as serverless functions. Due to the highly dynamic execution environment, multiple identical runs of the same serverless function can yield different performance, specifically in terms of end-to-end response latency. However, surprisingly, our analysis of serverless computing-related papers published in top-tier conferences highlights that the research community lacks awareness of the performance variance problem, with only 38.38% of these papers employing multiple runs for quantifying it. To further investigate, we analyze the performance of 72 serverless functions collected from these papers. Our findings reveal that the performance of these serverless functions can differ by up to 338.76% (44.28% on average) across different runs. Moreover, 61.11% of these functions produce unreliable performance results, with a low number of repetitions commonly employed in the serverless computing literature. Our study highlights a lack of awareness in the serverless computing community regarding the well-known performance variance problem in software engineering. The empirical results illustrate the substantial magnitude of this variance, emphasizing that ignoring the variance can affect research reproducibility and result reliability.