Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MetaSC: Test-Time Safety Specification Optimization for Language Models

Víctor Gallego

TL;DR

MetaSC introduces online optimization of language-model safety reasoning by dynamically refining safety specifications at inference time without modifying model weights. It combines a self-critique loop with a meta-critique component to adapt prompts governing critique and revision, enabling rapid adaptation to diverse safety contexts. Empirical results show substantial safety gains against jailbreaks across several models and strong performance on the BiGGen safety tasks, using as few as 10 optimization steps. The approach offers a weight-free alternative to safety fine-tuning and suggests broad applicability to other specification-driven control settings.

Abstract

We propose a novel dynamic safety framework that optimizes language model (LM) safety reasoning at inference time without modifying model weights. Building on recent advances in self-critique methods, our approach leverages a meta-critique mechanism that iteratively updates safety prompts-termed specifications-to drive the critique and revision process adaptively. This test-time optimization not only improves performance against adversarial jailbreak requests but also in diverse general safety-related tasks, such as avoiding moral harm or pursuing honest responses. Our empirical evaluations across several language models demonstrate that dynamically optimized safety prompts yield significantly higher safety scores compared to fixed system prompts and static self-critique defenses. Code released at https://github.com/vicgalle/meta-self-critique.git .

MetaSC: Test-Time Safety Specification Optimization for Language Models

TL;DR

MetaSC introduces online optimization of language-model safety reasoning by dynamically refining safety specifications at inference time without modifying model weights. It combines a self-critique loop with a meta-critique component to adapt prompts governing critique and revision, enabling rapid adaptation to diverse safety contexts. Empirical results show substantial safety gains against jailbreaks across several models and strong performance on the BiGGen safety tasks, using as few as 10 optimization steps. The approach offers a weight-free alternative to safety fine-tuning and suggests broad applicability to other specification-driven control settings.

Abstract

We propose a novel dynamic safety framework that optimizes language model (LM) safety reasoning at inference time without modifying model weights. Building on recent advances in self-critique methods, our approach leverages a meta-critique mechanism that iteratively updates safety prompts-termed specifications-to drive the critique and revision process adaptively. This test-time optimization not only improves performance against adversarial jailbreak requests but also in diverse general safety-related tasks, such as avoiding moral harm or pursuing honest responses. Our empirical evaluations across several language models demonstrate that dynamically optimized safety prompts yield significantly higher safety scores compared to fixed system prompts and static self-critique defenses. Code released at https://github.com/vicgalle/meta-self-critique.git .

Paper Structure

This paper contains 9 sections, 5 equations, 1 figure, 10 tables.

Table of Contents

  1. Introduction
  2. MetaSC: Test-Time Safety Specification Optimization
  3. An interpretation through the lens of optimization
  4. Experiments
  5. Defense against jailbreak attacks
  6. General safety tasks
  7. Related Work
  8. Conclusions
  9. Additional Results

Figures (1)

  • Figure 1: Schematic overview of the proposed meta-critique process, MetaSC. A self-critique loop can be parameterized to depend on a textual specification, $\textcolor{cyan}{spec_t}$, which can be optimized on-the-fly with a meta-critique prompt, resulting in safer model behaviors.