Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models

Ravi Ranjan, Utkarsh Grover, Xiaomin Lin, Agoritsa Polyzou

Abstract

Transformer based diffusion and vision-language models have achieved remarkable success; yet, efficiently removing undesirable or sensitive information without retraining remains a central challenge for model safety and compliance. We introduce Ratio-Aware Zero/One-step Optimized Retentive unlearning (RAZOR), a lightweight, model-agnostic unlearning framework that generalizes forgetting updates to coordinated multi-layer and multi-head edits within transformer backbones. RAZOR identifies the most important layers and attention heads by measuring how much they contribute to forgetting the target data while preserving useful knowledge. Then, it updates these parts of the model using a carefully regularized rule to avoid harming overall performance. The set of edited components grows gradually, ensuring precise unlearning without over-editing or damaging unrelated capabilities. We evaluate RAZOR on CLIP, Stable Diffusion, and vision-language models (VLMs) using widely adopted unlearning benchmarks covering identity, style, and object erasure tasks. Our results show that RAZOR achieves highly accurate and stable forgetting, even under quantization. This approach offers stronger retention and better efficiency than prior methods. Notably, it also operates significant faster than conventional techniques. These results demonstrate that RAZOR is a practical and scalable solution for safe, adaptive unlearning in transformer-based vision models.

RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models

Abstract

Transformer based diffusion and vision-language models have achieved remarkable success; yet, efficiently removing undesirable or sensitive information without retraining remains a central challenge for model safety and compliance. We introduce Ratio-Aware Zero/One-step Optimized Retentive unlearning (RAZOR), a lightweight, model-agnostic unlearning framework that generalizes forgetting updates to coordinated multi-layer and multi-head edits within transformer backbones. RAZOR identifies the most important layers and attention heads by measuring how much they contribute to forgetting the target data while preserving useful knowledge. Then, it updates these parts of the model using a carefully regularized rule to avoid harming overall performance. The set of edited components grows gradually, ensuring precise unlearning without over-editing or damaging unrelated capabilities. We evaluate RAZOR on CLIP, Stable Diffusion, and vision-language models (VLMs) using widely adopted unlearning benchmarks covering identity, style, and object erasure tasks. Our results show that RAZOR achieves highly accurate and stable forgetting, even under quantization. This approach offers stronger retention and better efficiency than prior methods. Notably, it also operates significant faster than conventional techniques. These results demonstrate that RAZOR is a practical and scalable solution for safe, adaptive unlearning in transformer-based vision models.
Paper Structure (33 sections, 15 equations, 6 figures, 8 tables, 1 algorithm)

This paper contains 33 sections, 15 equations, 6 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAZOR: Ratio-Aware Layer/Head Unlearning
  4. Preliminaries
  5. Ratio-Aware Layer/Head Selection
  6. Three-Loss Objective with Ratio Control
  7. One-/Few-Step Updates
  8. Iterative Growing of K
  9. Instantiation Across Model Families
  10. Experimental Setup
  11. Datasets and Data Construction
  12. Evaluation Metrics
  13. Results
  14. Results with CLIP
  15. Results with Stable Diffusion
  16. ...and 18 more sections

Figures (6)

  • Figure 1: RAZOR edits a subset of layers to unlearn selected visual identity while retaining others. And RAZOR operates in a model agnostic way across CLIP, vision language models(VLMs), and diffusion models.
  • Figure 2: Overview of the RAZOR unlearning pipeline across gradient computation, selective updates, and refinement.
  • Figure 3: Qualitative comparison of RAZOR and SLUG on SD-3 identity unlearning, showing stronger forgetting with better retention preservation.
  • Figure 4: Qualitative example of Identity unlearning on LLaVA-v1.6 with RAZOR shows that "Taylor Swift" was successfully unlearned without affecting any other identities/concepts.
  • Figure 5: Effect of learning rate change (1e-5 to 1e-20) on unlearning performance across M1–M5 metrics.
  • ...and 1 more figures