R-TOFU: Unlearning in Large Reasoning Models

TL;DR

R-TOFU presents the first benchmark for unlearning in Large Reasoning Models by attaching realistic chain-of-thought traces and introducing stepwise evaluation metrics. It demonstrates that unlearning must address reasoning traces in addition to final answers, showing that CoT-focused strategies like CoT-only and Reasoned IDK achieve stronger forgetting-utility trade-offs. The study also uncovers decoding-time vulnerabilities where approaches such as ZeroThink and LessThink can still reveal forgotten content, underscoring the need for multi-faceted evaluation. Collectively, the work establishes a systematic framework for reasoning-aware unlearning in LRMs and motivates further privacy-preserving developments.

Abstract

Large Reasoning Models (LRMs) embed private or copyrighted information not only in their final answers but also throughout multi-step chain-of-thought (CoT) traces, making reliable unlearning far more demanding than in standard LLMs. We introduce Reasoning-TOFU (R-TOFU), the first benchmark tailored to this setting. R-TOFU augments existing unlearning tasks with realistic CoT annotations and provides step-wise metrics that expose residual knowledge invisible to answer-level checks. Using R-TOFU, we carry out a comprehensive comparison of gradient-based and preference-optimization baselines and show that conventional answer-only objectives leave substantial forget traces in reasoning. We further propose Reasoned IDK, a preference-optimization variant that preserves coherent yet inconclusive reasoning, achieving a stronger balance between forgetting efficacy and model utility than earlier refusal styles. Finally, we identify a failure mode: decoding variants such as ZeroThink and LessThink can still reveal forgotten content despite seemingly successful unlearning, emphasizing the need to evaluate models under diverse decoding settings. Together, the benchmark, analysis, and new baseline establish a systematic foundation for studying and improving unlearning in LRMs while preserving their reasoning capabilities.

Figures (13)

• Figure 1: Overview of LRMs Unlearning. (a) Concept. Unlike standard LLMs, LRMs require unlearning both the final answer and the associated reasoning trace. (b) Unlearning Strategies. The top row illustrates gradient ascent-based strategies, while the bottom row presents preference optimization-based strategies. Red indicates forget information that the model is trained to suppress, while blue indicates non-forget responses (e.g., “I don’t know.” (IDK)) that replace forget content during training. (c) Interaction with Decoding. Although unlearning may appear successful under DefaultThink, decoding strategies like ZeroThink and LessThink, which suppress CoT generation, can still reveal forgotten content, indicating incomplete unlearning.
• Figure 2: Detailed Analysis of CFE Results in forget01. Step-wise ROUGE-L scores, step-wise Cosine Similarity, and LLM-as-Judge evaluations across four unlearning methods, showing reasoning trace unlearning efficacy.
• Figure 3: ROUGE scores of forget answers under different decoding strategies in the forget01 scenario. We plot ROUGE across unlearning epochs under DefaultThink, ZeroThink, and LessThink.
• Figure 4: Prompt used to rewrite fictitious author questions into real-author questions while preserving the original style.
• Figure 5: Prompt used to generate new chain-of-thought (CoT) traces for the original TOFU fictitious question-answer pairs, guided by real-author CoT examples.