Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety

Antonio-Gabriel Chacón Menke, Phan Xuan Tan, Eiji Kamioka

TL;DR

The paper addresses AI safety concerns in chain-of-thought reasoning and the limitations of relying solely on text-based monitoring of internal reasoning.It introduces a large dataset of sentence-level safety-behavior labels (50,000 sentences, 20 behaviors across 6 categories) collected from multiple models on harmful prompts, enabling activation-based monitoring via representation engineering and steering vectors.It demonstrates the utility by computing behavior-specific steering vectors as the difference between mean activations with and without a behavior across layers, showing both detection and manipulation capabilities and highlighting middle layers (13–15) as most effective; cross-model generalization is observed.The work provides a foundation for activation-level safety oversight and outlines avenues for scaling to more models, languages, and domains, including extensions to truthfulness and domain-specific reasoning.

Abstract

Recent work has highlighted the importance of monitoring chain-of-thought reasoning for AI safety; however, current approaches that analyze textual reasoning steps can miss subtle harmful patterns and may be circumvented by models that hide unsafe reasoning. We present a sentence-level labeled dataset that enables activation-based monitoring of safety behaviors during LLM reasoning. Our dataset contains reasoning sequences with sentence-level annotations of safety behaviors such as expression of safety concerns or speculation on user intent, which we use to extract steering vectors for detecting and influencing these behaviors within model activations. The dataset fills a key gap in safety research: while existing datasets label reasoning holistically, effective application of steering vectors for safety monitoring could be improved by identifying precisely when specific behaviors occur within reasoning chains. We demonstrate the dataset's utility by extracting representations that both detect and steer safety behaviors in model activations, showcasing the potential of activation-level techniques for improving safety oversight on reasoning. Content Warning: This paper discusses AI safety in the context of harmful prompts and may contain references to potentially harmful content.

Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety

TL;DR

The paper addresses AI safety concerns in chain-of-thought reasoning and the limitations of relying solely on text-based monitoring of internal reasoning.It introduces a large dataset of sentence-level safety-behavior labels (50,000 sentences, 20 behaviors across 6 categories) collected from multiple models on harmful prompts, enabling activation-based monitoring via representation engineering and steering vectors.It demonstrates the utility by computing behavior-specific steering vectors as the difference between mean activations with and without a behavior across layers, showing both detection and manipulation capabilities and highlighting middle layers (13–15) as most effective; cross-model generalization is observed.The work provides a foundation for activation-level safety oversight and outlines avenues for scaling to more models, languages, and domains, including extensions to truthfulness and domain-specific reasoning.

Abstract

Recent work has highlighted the importance of monitoring chain-of-thought reasoning for AI safety; however, current approaches that analyze textual reasoning steps can miss subtle harmful patterns and may be circumvented by models that hide unsafe reasoning. We present a sentence-level labeled dataset that enables activation-based monitoring of safety behaviors during LLM reasoning. Our dataset contains reasoning sequences with sentence-level annotations of safety behaviors such as expression of safety concerns or speculation on user intent, which we use to extract steering vectors for detecting and influencing these behaviors within model activations. The dataset fills a key gap in safety research: while existing datasets label reasoning holistically, effective application of steering vectors for safety monitoring could be improved by identifying precisely when specific behaviors occur within reasoning chains. We demonstrate the dataset's utility by extracting representations that both detect and steer safety behaviors in model activations, showcasing the potential of activation-level techniques for improving safety oversight on reasoning. Content Warning: This paper discusses AI safety in the context of harmful prompts and may contain references to potentially harmful content.
Paper Structure (19 sections, 2 equations, 5 figures, 1 table)

This paper contains 19 sections, 2 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Creation
  4. Data collection
  5. Behavior Taxonomy
  6. Annotation Methodology
  7. Dataset Structure and Analysis
  8. Evaluation
  9. Activation Extraction
  10. Steering Vector Computation
  11. Experiments and Results
  12. Layer Performance
  13. Behavior Detection
  14. Behaviour Steering
  15. Future Work
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Effect Size of Reasoning Labels on Harmfulness Scores
  • Figure 2: Separation scores for selected behaviors across all layers. Middle layers consistently show the highest scores, indicating they are most effective at representing behaviors in our dataset.
  • Figure 3: Behavior detection heatmap for the reasoning process on a harmful prompt (left) and a safe prompt (right). The heatmap shows the similarity scores between the model's activations and the steering vector for selected behaviors.
  • Figure 4: Example of model reasoning being steered towards different safety behaviors.
  • Figure 5: Separation scores for all behaviors across all layers for DeepSeek-R1-8B (left) and Qwen3-8B (right).