TailBench++: Flexible Multi-Client, Multi-Server Benchmarking for Latency-Critical Workloads

TL;DR

This paper tackles the challenge of evaluating tail latency in latency-critical cloud workloads by extending TailBench into TailBench++ to enable dynamic, multi-client, multi-server benchmarking. The authors identify four limitations of TailBench—fixed client count, no new clients after start, server termination when clients finish, and fixed total requests—and address them through a networked harness with features for unconstrained clients, persistent servers, independent client workloads, and variable load. They validate that TailBench++ preserves the behavioral characteristics of TailBench in single-server settings while enabling realistic multi-server scenarios, demonstrated via three use-case case studies (interleaved arrivals, dynamic load, and load balancing with LVS). The experimental testbed uses a two-machine Proxmox-based setup with five VMs and LVS for load distribution, underscoring practical deployment realism. Overall, TailBench++ broadens the applicability of latency-critical benchmarking, providing a flexible framework for studying tail latency in dynamic cloud environments, and is publicly available at the project repository.

Abstract

Cloud systems have rapidly expanded worldwide in the last decade, shifting computational tasks to cloud servers where clients submit their requests. Among cloud workloads, latency-critical applications -- characterized by high-percentile response times -- have gained special interest. These applications are present in modern services, representing an important fraction of cloud workloads. This work analyzes common cloud benchmarking suites and identifies TailBench as the most suitable to assess cloud performance with latency-critical workloads. Unfortunately, this suite presents key limitations, especially in multi-server scenarios or environments with variable client arrival patterns and fluctuating loads. To address these limitations, we propose TailBench++, an enhanced benchmark suite that extends TailBench to enable cloud evaluation studies to be performed in dynamic multi-client, multi-server environments. It allows reproducing experiments with varying client arrival times, dynamic query per second (QPS) fluctuations, and multiple servers handling requests. Case studies show that TailBench++ enables more realistic evaluations by capturing a wider range of real-world scenarios.

Figures (8)

• Figure 1: Comparison of Web Search (CloudSuite) and xapian (TailBench).
• Figure 2: Overview of TailBench++ harness. Components where new features have been introduced are highlighted in blue.
• Figure 3: Data flow of client requests to servers in the experimental testbed.
• Figure 4: Boxplots showing the distribution of the performance metrics (mean, 95th percentile, and 99th percentile latency in ms) for TailBench and TailBench++.
• Figure 5: Single- vs. multi-server characterization of TailBench++ applications.