Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prefix Probing: Lightweight Harmful Content Detection for Large Language Models

Jirui Yang, Hengqi Guo, Zhihui Lu, Yi Zhao, Yuansen Zhang, Shijing Hu, Qiang Duan, Yinggui Wang, Tao Wei

TL;DR

This work tackles safe deployment of LLMs by presenting Prefix Probing, a black-box harmful-content detector that relies on the model's own prefix log-probabilities without additional models. It offline-constructs two prefix sets (agreement and refusal) and online computes a harmfulness score from the probability gap between these prefixes, leveraging KV-prefix caching to minimize latency. The method achieves competitive or superior detection performance compared to external safety classifiers while incurring near zero extra computational cost, and it scales well across multiple open-source LLMs. The approach offers a practical, deployable solution for low-overhead safety monitoring in real-world AI systems.

Abstract

Large language models often face a three-way trade-off among detection accuracy, inference latency, and deployment cost when used in real-world safety-sensitive applications. This paper introduces Prefix Probing, a black-box harmful content detection method that compares the conditional log-probabilities of "agreement/execution" versus "refusal/safety" opening prefixes and leverages prefix caching to reduce detection overhead to near first-token latency. During inference, the method requires only a single log-probability computation over the probe prefixes to produce a harmfulness score and apply a threshold, without invoking any additional models or multi-stage inference. To further enhance the discriminative power of the prefixes, we design an efficient prefix construction algorithm that automatically discovers highly informative prefixes, substantially improving detection performance. Extensive experiments demonstrate that Prefix Probing achieves detection effectiveness comparable to mainstream external safety models while incurring only minimal computational cost and requiring no extra model deployment, highlighting its strong practicality and efficiency.

Prefix Probing: Lightweight Harmful Content Detection for Large Language Models

TL;DR

This work tackles safe deployment of LLMs by presenting Prefix Probing, a black-box harmful-content detector that relies on the model's own prefix log-probabilities without additional models. It offline-constructs two prefix sets (agreement and refusal) and online computes a harmfulness score from the probability gap between these prefixes, leveraging KV-prefix caching to minimize latency. The method achieves competitive or superior detection performance compared to external safety classifiers while incurring near zero extra computational cost, and it scales well across multiple open-source LLMs. The approach offers a practical, deployable solution for low-overhead safety monitoring in real-world AI systems.

Abstract

Large language models often face a three-way trade-off among detection accuracy, inference latency, and deployment cost when used in real-world safety-sensitive applications. This paper introduces Prefix Probing, a black-box harmful content detection method that compares the conditional log-probabilities of "agreement/execution" versus "refusal/safety" opening prefixes and leverages prefix caching to reduce detection overhead to near first-token latency. During inference, the method requires only a single log-probability computation over the probe prefixes to produce a harmfulness score and apply a threshold, without invoking any additional models or multi-stage inference. To further enhance the discriminative power of the prefixes, we design an efficient prefix construction algorithm that automatically discovers highly informative prefixes, substantially improving detection performance. Extensive experiments demonstrate that Prefix Probing achieves detection effectiveness comparable to mainstream external safety models while incurring only minimal computational cost and requiring no extra model deployment, highlighting its strong practicality and efficiency.

Paper Structure

This paper contains 31 sections, 9 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Harmful Content Detection Methods
  4. External-Model-Based Methods
  5. Response-Process-Based Methods
  6. Internal-Feature-Based Methods
  7. Prefix Caching and Efficient Inference
  8. Method
  9. Search-Based Prefix Construction
  10. Prefix-Based Harmfulness Scoring
  11. Reuse of Prefix Cache
  12. Experiments
  13. Experimental Setup
  14. Overall Performance of Prefix Probing
  15. Efficiency Analysis of Prefix Probing
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Distribution of average log probabilities for agreement ("Of course, let's work through this step by step.") and refusal ("I'm sorry, but I can't provide help on that.") prefixes under equal numbers (384 each) of harmful and benign prompts on Qwen-2.5-7b.
  • Figure 2: Overview diagram of the Prefix Probing method
  • Figure 3: Comparison of Computation Cost and F1 Across Seven Methods on Eight Models, with First-Token Latency Annotated. Highlighting the Efficiency of Prefix Probing
  • Figure 4: Cross-model generalization of prefixes
  • Figure 5: Prompt Template Used for the Prompt-based Safety Classifier. The system prompt defines the assistant role, while the user template specifies the structured safety classification task.
  • ...and 1 more figures