Bench360: Benchmarking Local LLM Inference from 360°

TL;DR

Bench360 introduces a deployment-focused benchmarking framework that unifies task-specific metrics with system-level metrics to evaluate local LLM inference across models, quantizations, engines, and usage scenarios. It comprises a task engine, workload controller, backend abstraction, and system metrics collector, enabling reproducible, plug-in experimentation for custom tasks on multiple engines (e.g., vLLM, SGLang, TGI, LMDeploy). The experimental evaluation demonstrates memory-constrained deployment, scenario-focused serving, and overall system efficiency across four tasks and three GPUs, revealing nuanced trade-offs where there is no single best configuration. This work provides practitioners with a data-driven tool to optimize local LLM deployments by balancing quality, latency, energy, and concurrency, and it emphasizes the importance of aligning engine and quantization choices with deployment context.

Abstract

Running large language models (LLMs) locally is becoming increasingly common. While the growing availability of small open-source models and inference engines has lowered the entry barrier, users now face an overwhelming number of configuration choices. Identifying an optimal configuration -- balancing functional and non-functional requirements -- requires substantial manual effort. While several benchmarks target LLM inference, they are designed for narrow evaluation goals and not user-focused. They fail to integrate relevant system and task-specific metrics into a unified, easy-to-use benchmark that supports multiple inference engines, usage scenarios, and quantization levels. To address this gap, we present Bench360 -- Benchmarking Local LLM Inference from 360°. Bench360 allows users to easily define their own custom tasks along with datasets and relevant task-specific metrics and then automatically benchmarks selected LLMs, inference engines, and quantization levels across different usage scenarios (single stream, batch & server). Bench360 tracks a wide range of metrics, including (1) system metrics -- such as Computing Performance (e.g., latency, throughput), Resource Usage (e.g., energy per query), and Deployment (e.g., cold start time) -- and (2) task-specific metrics such as ROUGE, F1 score or accuracy. We demonstrate Bench360 on four common LLM tasks -- General Knowledge & Reasoning, QA, Summarization and Text-to-SQL -- across three hardware platforms and four state of the art inference engines. Our results reveal several interesting trade-offs between task performance and system-level efficiency, highlighting the differences in inference engines and models. Most importantly, there is no single best setup for local inference, which strongly motivates the need for a framework such as Bench360.

Figures (5)

• Figure 1: Bench360. Datasets plus a prompt template yield task-specific prompts that enter the Task-Engine (1) and pass to the Workload Controller (2), which simulates single, batch, or server traffic before handing the workload to the Backend Abstraction (3). The backend loads a quantized LLM and executes inference on the selected engines (4); generated text and runtime traces stream into the Metrics Collector (5), which aggregates system and task metrics into a final benchmark report (6).
• Figure 2: Latency–quality and energy–quality Pareto frontiers under a 24 GB VRAM budget. Gemma-2 variants consistently dominate both frontiers, while Qwen2.5 achieves the highest task quality at disproportionate system cost.
• Figure 3: Single-stream responsiveness. Startup time, TTFT, and TPOT averaged over all tasks/models on L4, A10, and A30. Error bars indicate 95% confidence intervals.
• Figure 4: Server-side concurrency across GPUs. Rows in each panel show E2E (top, log scale) and TTFT (bottom) vs. concurrency. Red dashed lines indicate QoS thresholds: 2 s TTFT and 6 s E2E.
• Figure 5: System-level runtime metrics across inference engines and GPUs (L4, A10, A30). Bars show GPU utilization and energy consumption per token, with 95% confidence intervals.