TimeBill: Time-Budgeted Inference for Large Language Models

TL;DR

TimeBill tackles the challenge of meeting hard time budgets for autoregressive LLM inference by introducing a fine-grained response length predictor and a workload-guided end-to-end time estimator. These components enable a time-budgeted KV eviction mechanism that adapts to each input and budget, balancing task completion and response quality. The framework is validated with a concrete implementation on Qwen models, demonstrating superior average performance and competitive completion rates under various overrun strategies. The work offers a practical pathway for deploying LLMs in time-critical settings by tightly coupling predictive timing with runtime configuration.

Abstract

Large Language Models (LLMs) are increasingly deployed in time-critical systems, such as robotics, autonomous driving, embodied intelligence, and industrial automation, where generating accurate responses within a given time budget is crucial for decision-making, control, or safety-critical tasks. However, the auto-regressive generation process of LLMs makes it challenging to model and estimate the end-to-end execution time. Furthermore, existing efficient inference methods based on a fixed key-value (KV) cache eviction ratio struggle to adapt to varying tasks with diverse time budgets, where an improper eviction ratio may lead to incomplete inference or a drop in response performance. In this paper, we propose TimeBill, a novel time-budgeted inference framework for LLMs that balances the inference efficiency and response performance. To be more specific, we propose a fine-grained response length predictor (RLP) and an execution time estimator (ETE) to accurately predict the end-to-end execution time of LLMs. Following this, we develop a time-budgeted efficient inference approach that adaptively adjusts the KV cache eviction ratio based on execution time prediction and the given time budget. Finally, through extensive experiments, we demonstrate the advantages of TimeBill in improving task completion rate and maintaining response performance under various overrun strategies.

Figures (8)

• Figure 1: An example of different inference strategies. The vanilla inference may overrun and miss the deadline, resulting in incomplete output. The As-Fast-As-Possible (AFAP) strategy will degrade the response performance, while time-budgeted inference improves the response performance under timing constraints.
• Figure 2: The overview of the TimeBill framework.
• Figure 3: The overview of the proposed fine-grained response length predictor (RLP).
• Figure 4: The timeline of TimeBill, where incoming arrows represent inputs (e.g., $x_1, N_{x_1}$) , and outgoing arrows represent outputs (e.g., $\hat{\mathbf{y}}_1, \alpha_1^*$).
• Figure 5: Fitted curves for estimating $\hat{t}_{\text{prefill-phase}}, \hat{t}_{\text{decoding-step}}$.