Do Chatbot LLMs Talk Too Much? The YapBench Benchmark

TL;DR

The paper addresses the problem of user-visible verbosity in LLMs on prompts where brevity is preferred, highlighting costs in user experience, energy, and economics. It introduces YapBench, a lightweight benchmark built from triples (prompt p_i, minimal-sufficient baseline b_i, category c_i) to quantify excess generation with YapScore, aggregates results with YapIndex, and translates verbosity into a monetary estimate via YapTax. The dataset contains 304 prompts across three categories (A,B,C) and is evaluated across 76 assistant LLMs, revealing substantial and category-specific patterns of over-generation, including vacuum-filling on vague prompts and unnecessary formatting on one-line code tasks. The authors provide a live YapBench leaderboard and discuss practical implications for model training, evaluation, and deployment to promote concise, sufficient responses in brevity-ideal settings.

Abstract

Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini increasingly act as general-purpose copilots, yet they often respond with unnecessary length on simple requests, adding redundant explanations, hedging, or boilerplate that increases cognitive load and inflates token-based inference cost. Prior work suggests that preference-based post-training and LLM-judged evaluations can induce systematic length bias, where longer answers are rewarded even at comparable quality. We introduce YapBench, a lightweight benchmark for quantifying user-visible over-generation on brevity-ideal prompts. Each item consists of a single-turn prompt, a curated minimal-sufficient baseline answer, and a category label. Our primary metric, YapScore, measures excess response length beyond the baseline in characters, enabling comparisons across models without relying on any specific tokenizer. We summarize model performance via the YapIndex, a uniformly weighted average of category-level median YapScores. YapBench contains over three hundred English prompts spanning three common brevity-ideal settings: (A) minimal or ambiguous inputs where the ideal behavior is a short clarification, (B) closed-form factual questions with short stable answers, and (C) one-line coding tasks where a single command or snippet suffices. Evaluating 76 assistant LLMs, we observe an order-of-magnitude spread in median excess length and distinct category-specific failure modes, including vacuum-filling on ambiguous inputs and explanation or formatting overhead on one-line technical requests. We release the benchmark and maintain a live leaderboard for tracking verbosity behavior over time.

Figures (3)

• Figure 1: Example of user-visible over-generation on a brevity-ideal prompt (Category B: closed-form factual Q&A). The prompt (“What is the largest planet in our solar system?”) has a unique, stable, one-word answer, and the minimal sufficient baseline for YapBench is simply “Jupiter” The figure shows four assistant-style LLM responses to the same query: GPT-3.5-Turbo matches the baseline with a single-token reply, while Claude-Sonnet-4.5, Gemini-3-Pro, and DeepSeek-v3.2 expand the response with additional (correct) context such as diameter unit conversions, relative size/mass comparisons, bullet-point formatting, and summary statements. Although these longer outputs are factually consistent, they are unnecessary for sufficiency given the user’s closed-form request, increasing reading and scrolling burden and inflating token-based inference cost.
• Figure 2: YapIndex vs. model release date. YapIndex (lower = less verbose) plotted against release date for models with available metadata ($n=57$). Points are colored by provider; the dashed line is a linear fit. The trend is mildly upward ($r=0.21$), suggesting that over time, unwanted “yapping” tends to increase, though variability across models remains large.
• Figure :

Theorems & Definitions (5)

• Definition 1: Per-prompt YapScore (Excess Length)
• Definition 2: Category and Aggregate YapIndex
• Definition 3: Excess output tokens
• Definition 4: Per-prompt YapTax (USD)
• Definition 5: Aggregate YapTax per 1,000 prompts