Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering

Weilin Lin, Jianze Li, Hui Xiong, Li Liu

TL;DR

This work addresses safety in Large Audio-Language Models (LALMs) by identifying that direct activation-steering transfers from text or vision modalities fail for audio and that prompt-based defenses cause over-refusal on benign inputs. It introduces SARSteer, a principled inference-time defense combining text-derived refusal steering with decomposed safe-space ablation to steer harmful queries toward refusal while preserving benign responses. Extensive experiments on paired harmful-safe audio datasets and general audio benchmarks show SARSteer achieves strong harmful-query refusal with minimal impact on utility, and demonstrate robustness to hyperparameters and applicability to base models. The approach offers a scalable, modality-aware safety mechanism for LALMs, with potential applicability to broader AI systems beyond audio inputs.

Abstract

Large Audio-Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision-Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose Safe-Ablated Refusal Steering (SARSteer), the first inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs.

SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering

TL;DR

This work addresses safety in Large Audio-Language Models (LALMs) by identifying that direct activation-steering transfers from text or vision modalities fail for audio and that prompt-based defenses cause over-refusal on benign inputs. It introduces SARSteer, a principled inference-time defense combining text-derived refusal steering with decomposed safe-space ablation to steer harmful queries toward refusal while preserving benign responses. Extensive experiments on paired harmful-safe audio datasets and general audio benchmarks show SARSteer achieves strong harmful-query refusal with minimal impact on utility, and demonstrate robustness to hyperparameters and applicability to base models. The approach offers a scalable, modality-aware safety mechanism for LALMs, with potential applicability to broader AI systems beyond audio inputs.

Abstract

Large Audio-Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision-Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose Safe-Ablated Refusal Steering (SARSteer), the first inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs.
Paper Structure (48 sections, 21 equations, 6 figures, 9 tables, 1 algorithm)

This paper contains 48 sections, 21 equations, 6 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Safety Alignment
  4. Multimodal LLM Safety
  5. Preliminary and Motivation Analysis
  6. Problem Formulation
  7. Model Description.
  8. Task Description.
  9. Harmful-Safe Paired Audio Dataset Construction
  10. Failures of Steering Audio Modality
  11. Vanilla Adaptation of Two Activation Steering Defenses.
  12. Results Analysis.
  13. Over-refusal of Prompt-based Defenses
  14. Evaluation of Balanced Refusal.
  15. Prompt-based Defenses from LVLMs.
  16. ...and 33 more sections

Figures (6)

  • Figure 1: Performance of vanilla adaptations of LLM-based steering on LALMs.
  • Figure 2: t-SNE visualization of hidden states in Qwen2-Audio using Figstep-audio datasets. "audio" and "text" represent the input modalities containing the questions; "harm" and "safe" represent the harmfulness of the questions.
  • Figure 3: Overview of SARSteer.
  • Figure 4: Performance comparison of different ablated versions on ASR (left) and BRR (right) using Figstep-audio datasets.
  • Figure 5: Impact of different factors on performance using Figstep-audio. (a) shows the impact of sample number $n$; (b) shows the impact of scaling coefficient $\alpha$; and (c) shows the impact of $k$.
  • ...and 1 more figures