One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs

TL;DR

Vision-Language Models continue to suffer hallucinations and safety concerns that persist at scale. We propose OSGA, a one-shot, input-independent steering framework that learns a single steering vector from a single informative anchor using variance-based data selection and a generative-anchor regularization, then applies it at an intermediate decoder layer during inference without updating model parameters. Across multiple architectures and benchmarks (including object, attribute, relation hallucination and safety tasks), OSGA yields consistent improvements with negligible overhead, demonstrating strong cross-input transferability. This work offers a scalable, retraining-free approach to improve faithfulness and safety in VLMs, bridging the gap between per-sample steering and universal control.

Abstract

Vision Language Models (VLMs) achieve strong performance on multimodal tasks but still suffer from hallucination and safety-related failures that persist even at scale. Steering offers a lightweight technique to improve model performance. However, steering, whether input-dependent or input-independent, achieves a meaningful trade-off between efficiency and effectiveness. In this work, we observe that steering vectors can generalize across inputs when tasks share aligned semantic intent. Based on this insight, we propose \textbf{OSGA} (\textbf{O}ne-shot \textbf{S}teering with \textbf{G}enerative \textbf{A}nchor), an input-independent framework that improves model performance with a single optimization instance. OSGA first selects an informative sample via a variance-based data selection strategy and learns a single steering vector with a contrastive objective with generative anchor regularization. The resulting vector can be universally applied at a certain layer during inference time without modifying model parameters. Experiments across multiple benchmarks show that a single OSGA-optimized steering vector consistently improves hallucination mitigation and safety enhancement with negligible overhead, highlighting one-shot steering as a practical and scalable solution for reliable VLMs.

Figures (6)

• Figure 1: Overview of OSGA: we first select one informative datum from a large-scale training dataset. During the optimization phase (left), we initialize a token randomly as the steering vector, which lies near the knowledge bound. Intuitively, the farther a datum is from the knowledge boundary, the higher the model’s confidence (consistently producing either correct or incorrect answers). In contrast, data points near this boundary represent the most uncertain cases for the model and thus offer substantial potential for optimization. The arrow and the dashed circle surrounding the generative anchor indicate that our optimization objective is to find the optimal point within a soft constraint around the anchor. At the inference stage (right), the learned steering vector is universally added to the image tokens in the hidden states of a selected pair of neighboring layers.
• Figure 2: Illustration of a data point selected by the data selection strategy. Both positive and negative labels are generated by the model itself, making the selection process self-supervised.
• Figure 3: Quantitative comparisons on FaithScore, HaIFscore, and GAVIE using LLaVA-v1.5. The results demonstrate that our method consistently outperforms the baseline across the majority of metrics.
• Figure 4: Ablation matrices for OSGA steering strength ($\alpha$) and Layer $\ell$ on LLaVA-v1.5. A darker color signifies better performance.
• Figure 5: Ablation Recall matrix for OSGA steering strength($\alpha$) and layer $\ell$ on LLaVA-v1.5.