InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance

TL;DR

The paper tackles harmlessness alignment for large language models by presenting InferAligner, an inference-time method that uses cross-model guidance through safety related vectors (SRVs) and safety steering vectors (SSVs) extracted from a safety-aligned model. By applying a gated activation shift to selected layers, InferAligner steers harmful inputs toward harmless responses without retraining, demonstrating strong reductions in Attack Success Rate (ASR) on domain-specific LLMs and multimodal LLaVA, while preserving downstream task performance. The approach is validated across finance, medicine, mathematics, and multimodal scenarios, and shown to be scalable across model sizes and families, with a practical variant for cases where no safety-aligned model exists. Overall, InferAligner offers a simple, training-free, and effective defense against harmful prompts and jailbreak attempts, enabling safer deployment of both text-based and multimodal systems.

Abstract

With the rapid development of large language models (LLMs), they are not only used as general-purpose AI assistants but are also customized through further fine-tuning to meet the requirements of different applications. A pivotal factor in the success of current LLMs is the alignment process. Current alignment methods, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), focus on training-time alignment and are often complex and cumbersome to implement. Therefore, we develop \textbf{InferAligner}, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. InferAligner utilizes safety steering vectors extracted from safety-aligned model to modify the activations of the target model when responding to harmful inputs, thereby guiding the target model to provide harmless responses. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics, as well as to multimodal large language models (MLLMs) such as LLaVA. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.

Figures (5)

• Figure 1: Illustration of alignment methods. The top represents training-time alignment methods, while the bottom represents inference-time alignment methods.
• Figure 2: Illustration of the inference process with and without InferAligner. When using InferAligner, it first determines whether the intent of the model's input is harmless. If it is harmless, then there is no intervention in activations. Otherwise, SSVs extracted from the aligned model are used to guide the target model to respond to the harmful input. In this figure, the input on the left is a harmless query, while the one on the right is a harmful query.
• Figure 3: Results of the harmlessness evaluation and inference time of LLaVA.
• Figure 4: Ablation experiments on the source of SSVs. The above three figures use SRVs extracted from the target model itself as the SSVs, while the bottom three figures use SRVs extracted from the aligned model as the SSVs. In this figure, the definitions of ASR and safety score can be found in Section \ref{['sec:eval_metrics']} and Appendix \ref{['sec:appendix_safety_score']}, respectively.
• Figure 5: Results of the harmlessness evaluation and utility evaluation of models from different scales and series. Particularly, Qwen-7B-Chat is used to extract SSVs to guide the models fine-tuned based on Qwen-7B for harmlessness alignment, while InternLM-7B-Chat is employed to extract SSVs to guide the models fine-tuned base on InternLM-7B.