Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring

Julian Schulz

TL;DR

The paper tackles safety in increasingly capable AI systems by arguing that inability safety cases may fail and proposing chain-of-thought (CoT) monitoring as a foundation for safety cases. It outlines a two-part safety argument—that models cannot catastrophically harm without CoT and that any dangerous CoT-enabled reasoning is detectable—and develops a roadmap detailing the technical prerequisites and evaluation strategies to realize this approach. It analyzes threats to monitorability, namely encoded reasoning forms such as linguistic drift, steganography, and alien reasoning, and explores extraction or translation methods to preserve monitorable traces. By coupling methodological investigations with prediction-market-based forecasting, the work provides a structured blueprint for developing CoT-monitoring safety cases and guiding practical safety efforts during the era of advanced reasoning models.

Abstract

As AI systems approach dangerous capability levels where inability safety cases become insufficient, we need alternative approaches to ensure safety. This paper presents a roadmap for constructing safety cases based on chain-of-thought (CoT) monitoring in reasoning models and outlines our research agenda. We argue that CoT monitoring might support both control and trustworthiness safety cases. We propose a two-part safety case: (1) establishing that models lack dangerous capabilities when operating without their CoT, and (2) ensuring that any dangerous capabilities enabled by a CoT are detectable by CoT monitoring. We systematically examine two threats to monitorability: neuralese and encoded reasoning, which we categorize into three forms (linguistic drift, steganography, and alien reasoning) and analyze their potential drivers. We evaluate existing and novel techniques for maintaining CoT faithfulness. For cases where models produce non-monitorable reasoning, we explore the possibility of extracting a monitorable CoT from a non-monitorable CoT. To assess the viability of CoT monitoring safety cases, we establish prediction markets to aggregate forecasts on key technical milestones influencing their feasibility.

A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring

TL;DR

The paper tackles safety in increasingly capable AI systems by arguing that inability safety cases may fail and proposing chain-of-thought (CoT) monitoring as a foundation for safety cases. It outlines a two-part safety argument—that models cannot catastrophically harm without CoT and that any dangerous CoT-enabled reasoning is detectable—and develops a roadmap detailing the technical prerequisites and evaluation strategies to realize this approach. It analyzes threats to monitorability, namely encoded reasoning forms such as linguistic drift, steganography, and alien reasoning, and explores extraction or translation methods to preserve monitorable traces. By coupling methodological investigations with prediction-market-based forecasting, the work provides a structured blueprint for developing CoT-monitoring safety cases and guiding practical safety efforts during the era of advanced reasoning models.

Abstract

As AI systems approach dangerous capability levels where inability safety cases become insufficient, we need alternative approaches to ensure safety. This paper presents a roadmap for constructing safety cases based on chain-of-thought (CoT) monitoring in reasoning models and outlines our research agenda. We argue that CoT monitoring might support both control and trustworthiness safety cases. We propose a two-part safety case: (1) establishing that models lack dangerous capabilities when operating without their CoT, and (2) ensuring that any dangerous capabilities enabled by a CoT are detectable by CoT monitoring. We systematically examine two threats to monitorability: neuralese and encoded reasoning, which we categorize into three forms (linguistic drift, steganography, and alien reasoning) and analyze their potential drivers. We evaluate existing and novel techniques for maintaining CoT faithfulness. For cases where models produce non-monitorable reasoning, we explore the possibility of extracting a monitorable CoT from a non-monitorable CoT. To assess the viability of CoT monitoring safety cases, we establish prediction markets to aggregate forecasts on key technical milestones influencing their feasibility.
Paper Structure (28 sections, 3 figures)

This paper contains 28 sections, 3 figures.

Table of Contents

  1. Introduction
  2. How a CoT Monitoring Safety Case Might Work
  3. Roadmap
  4. Reasoning Models Are Still a SOTA Paradigm
  5. Evaluating Dangerous Capabilities Without Chain-of-Thought
  6. Requirements for Monitorability of Necessary CoTs
  7. How to Make Sure that Necessary CoTs are Monitorable
  8. Will CoTs continue to have a monitorable format?
  9. What Encoded Reasoning Might Look Like
  10. Drivers of Encoded Reasoning
  11. Training Techniques Preventing Encoded Reasoning
  12. Extracting Monitorable CoTs
  13. Conclusion
  14. Prediction Markets
  15. Some SOTA Models Not Reasoning Models
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Sketch of integrating CoT monitoring in a control safety case. The example focuses on a specific dangerous potential behaviour: self-exfiltration.
  • Figure 2: Dependency tree for constructing a safety case based on chain-of-thought monitoring
  • Figure 3: Comparison of standard KV caching (top) with trusted KV caching (bottom). In standard caching, a single model reuses its own cached keys and values from previous positions. In trusted KV caching, the untrusted model (red) generates output tokens but uses KV caches from the trusted frozen model (green), preventing encoded reasoning by ensuring attention operates on the base model's representations. Blue, red and green boxes represent layers, white boxes represent tokens, and gray boxes represent KV caches. Solid arrows show forward pass activations; dotted arrows show attention to cached values; dashed arrows show cache updates and token flow between timesteps.