Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation

Hossein Hosseini Kasnavieh, Gholamreza Haffari, Chris Leckie, Adel N. Toosi

TL;DR

IntroLM enables causal language models to predict their own output quality during the prefilling phase by appending special [CPX] introspection tokens and training with token-conditional LoRA, avoiding any change to the generation process. It achieves strong complexity-prediction performance on long-context question answering and enables more cost-efficient multi-model routing by reducing unnecessary large-model calls and latency at matched reliability. The method preserves backbone behavior, reuses prefilling representations, and requires only lightweight adapters, facilitating practical deployment in retrieval-augmented and multi-model systems. Overall, IntroLM demonstrates how prefilling-time introspection can improve both the accuracy and efficiency of large-scale language-model pipelines.

Abstract

A major challenge for the operation of large language models (LLMs) is how to predict whether a specific LLM will produce sufficiently high-quality output for a given query. Existing approaches rely on external classifiers, most commonly BERT based models, which suffer from limited context windows, constrained representational capacity, and additional computational overhead. We propose IntroLM, a method that enables causal language models to predict their own output quality during the prefilling phase without affecting generation using introspective tokens. By introducing token conditional LoRA that activates only for the introspective token, the model learns to predict the output quality for a given query while preserving the original backbone behavior and avoiding external evaluators. On question answering benchmarks, IntroLM applied to Qwen3 8B achieves a ROC AUC of 90 precent for success prediction, outperforming a DeBERTa classifier by 14 precent. When integrated into multi model routing systems, IntroLM achieves superior cost performance tradeoffs, reducing latency by up to 33 precent and large model usage by up to 50 precent at matched reliability.

IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation

TL;DR

IntroLM enables causal language models to predict their own output quality during the prefilling phase by appending special [CPX] introspection tokens and training with token-conditional LoRA, avoiding any change to the generation process. It achieves strong complexity-prediction performance on long-context question answering and enables more cost-efficient multi-model routing by reducing unnecessary large-model calls and latency at matched reliability. The method preserves backbone behavior, reuses prefilling representations, and requires only lightweight adapters, facilitating practical deployment in retrieval-augmented and multi-model systems. Overall, IntroLM demonstrates how prefilling-time introspection can improve both the accuracy and efficiency of large-scale language-model pipelines.

Abstract

A major challenge for the operation of large language models (LLMs) is how to predict whether a specific LLM will produce sufficiently high-quality output for a given query. Existing approaches rely on external classifiers, most commonly BERT based models, which suffer from limited context windows, constrained representational capacity, and additional computational overhead. We propose IntroLM, a method that enables causal language models to predict their own output quality during the prefilling phase without affecting generation using introspective tokens. By introducing token conditional LoRA that activates only for the introspective token, the model learns to predict the output quality for a given query while preserving the original backbone behavior and avoiding external evaluators. On question answering benchmarks, IntroLM applied to Qwen3 8B achieves a ROC AUC of 90 precent for success prediction, outperforming a DeBERTa classifier by 14 precent. When integrated into multi model routing systems, IntroLM achieves superior cost performance tradeoffs, reducing latency by up to 33 precent and large model usage by up to 50 precent at matched reliability.
Paper Structure (44 sections, 9 equations, 7 figures, 5 tables)

This paper contains 44 sections, 9 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Formulation
  5. Prefilling and Decoding Stages of LLMs
  6. Introspective Language Models (IntroLM)
  7. Training via Frozen Weights for [CPX]
  8. Training via Token-Conditional LoRA
  9. Impact on projections and [CPX] adaptation.
  10. IntroLM-Based Routing
  11. Routing Problem
  12. Our Routing Approach
  13. Routing Metrics
  14. Reliability
  15. Large-model call rate
  16. ...and 29 more sections

Figures (7)

  • Figure 1: Overview of IntroLM. Complexity logits are computed during prefilling using [CPX] tokens, while generation remains unchanged.
  • Figure 2: Training via Frozen Weights for [CPX]. Only the classifier head and [CPX] token embedding are trainable.
  • Figure 3: An illustration of token-conditional LoRA in any projection. The updated LoRA weights are effectively masked for non-[CPX] tokens.
  • Figure 4: ROC curves comparing IntroLM (solid blue curve) and BERT-based baselines on (a) General QA (b) HotpotQA (c) daily-style chat prompts (LMSYS- Chat-1M). IntroLM achieves consistently stronger separability between simple and complex prompts.
  • Figure 5: Routing trade-offs on General QA and HotpotQA. IntroLM (solid blue curve) consistently achieves lower latency and fewer large-model invocations at matched reliability.
  • ...and 2 more figures