CASTILLO: Characterizing Response Length Distributions of Large Language Models

TL;DR

CASTILLO addresses the challenge of unpredictable LLM response lengths that hamper resource planning in production systems by providing a large, open dataset of response-length distributions across 13 open-source LLMs and 7 instruction-following datasets. The approach generates 10 completions per ⟨prompt, model⟩ pair under fixed decoding settings, computing mean, std-dev, and percentiles, and also flags text degeneration to support sanitized analyses. Key contributions include a rigorous characterization of inter- and intra-model variability, a dual-subset dataset (sanitized and degeneration-only), and public release of data and tooling to enable predictive length estimation, benchmarking, and system-level simulations. The work enables researchers to build proactive scheduling models, compare generation behaviors across models, and study degeneration patterns, with practical impact on LLM serving efficiency and reliability.

Abstract

Efficiently managing compute resources for Large Language Model (LLM) inference remains challenging due to the inherently stochastic and variable lengths of autoregressive text generation. Accurately estimating response lengths in advance enables proactive resource allocation, yet existing approaches either bias text generation towards certain lengths or rely on assumptions that ignore model- and prompt-specific variability. We introduce CASTILLO, a dataset characterizing response length distributions across 13 widely-used open-source LLMs evaluated on seven distinct instruction-following corpora. For each $\langle$prompt, model$\rangle$ sample pair, we generate 10 independent completions using fixed decoding hyper-parameters, record the token length of each response, and publish summary statistics (mean, std-dev, percentiles), along with the shortest and longest completions, and the exact generation settings. Our analysis reveals significant inter- and intra-model variability in response lengths (even under identical generation settings), as well as model-specific behaviors and occurrences of partial text degeneration in only subsets of responses. CASTILLO enables the development of predictive models for proactive scheduling and provides a systematic framework for analyzing model-specific generation behaviors. We publicly release the dataset and code to foster research at the intersection of generative language modeling and systems.

Figures (17)

• Figure 1: LLM response lengths exhibit significant inter- and intra-model variability. Points depict the mean response length for a ⟨prompt, model⟩ pair; shaded regions depict ±1 standard deviation over 10 independent responses under fixed decoding parameters. Significant variation both among different models and within the 10 sampled responses, even within the same model.
• Figure 2: Transformer-based LLM inference in production is divided into a fast and compute intensive phase (prefill) and a time-consuming and memory intensive phase (decode).
• Figure 3: Both the average mean and std-dev across models and dataset can vary by several hundreds of tokens. Heatmaps depict the average of both the mean (\ref{['subfig:heatmap_mean']}) and std-dev (\ref{['subfig:heatmap_std']}) of response lengths across models and datasets for all generations performed.
• Figure 4: LLMs exhibit substantial variation in their distribution of both mean and std-dev of their responses. Figure\ref{['subfig:boxplot_mean-resp-length']} shows large disparities across both models and datasets, even under identical generation settings.
• Figure 5: High coefficient of variation between responses for the same prompt from an LLM. Figure depicts average coefficients of variation (std-dev normalized by the mean) across datasets and models ranging from $7\%$ to up to $45\%$. Hence, the std-dev can reach up to half of the value of the average response length in the batch.