TUNA: Tuning Unstable and Noisy Cloud Applications

TL;DR

The paper tackles the problem of autotuning cloud applications under performance variability and unstable configurations. It introduces TUNA, a tuning framework that adds multi-fidelity sampling, unstable configuration detection, and a noise-adjusting model to de-noise measurements while preserving compatibility with existing optimizers. Through extensive experiments across workloads, regions, hardware, and systems, TUNA demonstrates faster convergence, reduced variability, and fewer unstable deployments compared with traditional single-machine sampling and equal-cost baselines. The approach promises practical impact by enabling more reliable and efficient cloud autotuning and by releasing associated datasets and artifacts for the community.

Abstract

Autotuning plays a pivotal role in optimizing the performance of systems, particularly in large-scale cloud deployments. One of the main challenges in performing autotuning in the cloud arises from performance variability. We first investigate the extent to which noise slows autotuning and find that as little as $5\%$ noise can lead to a $2.5$x slowdown in converging to the best-performing configuration. We measure the magnitude of noise in cloud computing settings and find that while some components (CPU, disk) have almost no performance variability, there are still sources of significant variability (caches, memory). Furthermore, variability leads to autotuning finding unstable configurations. As many as $63.3\%$ of the configurations selected as "best" during tuning can have their performance degrade by $30\%$ or more when deployed. Using this as motivation, we propose a novel approach to improve the efficiency of autotuning systems by (a) detecting and removing outlier configurations and (b) using ML-based approaches to provide a more stable true signal of de-noised experiment results to the optimizer. The resulting system, TUNA (Tuning Unstable and Noisy Cloud Applications) enables faster convergence and robust configurations. Tuning postgres running mssales, an enterprise production workload, we find that TUNA can lead to $1.88$x lower running time on average with $2.58x$ lower standard deviation compared to traditional sampling methodologies.

Figures (22)

• Figure 1: Modern auto-tuning system design LlamatuneOttertuneCDBTune
• Figure 2: Optimizer Rate of Convergence for epinions workload, running on PostgreSQL 16.1 at various levels of noise.
• Figure 3: The variance of PostgreSQL and Redis benchmarks. Relative performance is relative to the mean performance seen within a region and VM SKU type. Higher is better.
• Figure 4: The variance of benchmarks targeting CPU, Disk, Memory, the OS, and CPU cache. Relative performance is relative to the mean performance seen. Higher is better.
• Figure 5: The performance of initialization and of best-performing configurations (picked by the optimizer) during training (left) and when deployed on new nodes (right).