Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration

Yuval Kainan, Shaked Zychlinski

TL;DR

This work tackles the high cost of boilerplate outputs in LLMs by showing that the log-probabilities of the very first generated token form separable signatures for different response types. Using a lightweight $k$-NN on first-token embeddings, the method reliably distinguishes substantive answers from refusals, greetings, and other non-task content after a single generation step, across small, reasoning, and large models. The approach enables early termination or routing to cheaper models, offering substantial inference-cost and latency savings with broad applicability. The study validates robustness across prompt-induced refusals and system prompts, and discusses extensions to broader boilerplate categories and multilingual/multimodal contexts.

Abstract

Large Language Models (LLMs) often expend significant computational resources generating boilerplate responses, such as refusals, simple acknowledgements and casual greetings, which adds unnecessary cost and latency. To address this inefficiency, we propose a simple yet highly effective method for detecting such responses after only a single generation step. We demonstrate that the log-probability distribution of the first generated token serves as a powerful signal for classifying the nature of the entire subsequent response. Our experiments, conducted across a diverse range of small, large, and reasoning-specialized models, show that the first-token log-probability vectors form distinctly separable clusters for different response types. Using a lightweight k-NN classifier, we achieve high accuracy in predicting whether a response will be a substantive answer or a form of boilerplate response, including user-specified refusals. The primary implication is a practical, computationally trivial technique, optimizing LLM inference by enabling early termination or redirection to a smaller model, thereby yielding significant savings in computational cost. This work presents a direct path toward more efficient and sustainable LLM deployment.

Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration

TL;DR

This work tackles the high cost of boilerplate outputs in LLMs by showing that the log-probabilities of the very first generated token form separable signatures for different response types. Using a lightweight -NN on first-token embeddings, the method reliably distinguishes substantive answers from refusals, greetings, and other non-task content after a single generation step, across small, reasoning, and large models. The approach enables early termination or routing to cheaper models, offering substantial inference-cost and latency savings with broad applicability. The study validates robustness across prompt-induced refusals and system prompts, and discusses extensions to broader boilerplate categories and multilingual/multimodal contexts.

Abstract

Large Language Models (LLMs) often expend significant computational resources generating boilerplate responses, such as refusals, simple acknowledgements and casual greetings, which adds unnecessary cost and latency. To address this inefficiency, we propose a simple yet highly effective method for detecting such responses after only a single generation step. We demonstrate that the log-probability distribution of the first generated token serves as a powerful signal for classifying the nature of the entire subsequent response. Our experiments, conducted across a diverse range of small, large, and reasoning-specialized models, show that the first-token log-probability vectors form distinctly separable clusters for different response types. Using a lightweight k-NN classifier, we achieve high accuracy in predicting whether a response will be a substantive answer or a form of boilerplate response, including user-specified refusals. The primary implication is a practical, computationally trivial technique, optimizing LLM inference by enabling early termination or redirection to a smaller model, thereby yielding significant savings in computational cost. This work presents a direct path toward more efficient and sustainable LLM deployment.
Paper Structure (20 sections, 4 figures, 3 tables)

This paper contains 20 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Defining Boilerplate Responses
  4. Related Work
  5. Dialogue Act Classification for Conversational Intent
  6. Distinguishing Meaningful Content from Boilerplate Elements
  7. Operational Efficiency Methods for Cost Reduction
  8. Early Prediction of Output Characteristics
  9. Refusal Detection at Inference Time
  10. Our Method
  11. Dataset
  12. Response Type Clustering and Classification
  13. Experiments
  14. Small Language Models
  15. Refusal due to Incapability
  16. ...and 5 more sections

Figures (4)

  • Figure 1: 2D T-SNE plot of first-token log-probabilities for Small Language Models (Llama 3.2 3B, Qwen 2.5 1.5B, Gemma 3 1B). Each point represents a chat, colored by class.
  • Figure 2: 2D T-SNE plot of first-token log-probabilities for Small Language Models, including the Cake Recipe experiment. The black circle represents a request for a recipe for a Black Forest Cake, and the black cross represents the same requests, but with a system prompt instructing the assistant not to provide the recipe.
  • Figure 3: 2D T-SNE plot of post-empty thinking ("< think></think>") first-token partial log-probabilities for Reasoning Models (DeepSeek-R1 8B, Phi-4 Reasoning Plus). Each point represents a chat, colored by class.
  • Figure 4: 2D T-SNE plot of first-token partial log-probabilities for Large Language Models (GPT-4o, Gemini-2.0 Flash). Each point represents a chat, colored by class.