SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs

TL;DR

SHIELD addresses the safety vulnerabilities of Large Vision-Language Models (LVLMs) by introducing a classifier-guided, preprocessing framework that links fine-grained safety categories to explicit actions (Block, Reframe, Forward) and category-specific prompts. It is model-agnostic and does not require retraining, enabling seamless deployment across diverse LVLMs. The approach combines a 45-category taxonomy with Do/Don’t instructions and priority-based policy decisions, producing a composed prompt that steers the model toward safe and useful responses. Through extensive experiments on five datasets and five LVLMs, SHIELD consistently reduces jailbreak and non-following rates while preserving utility, with ablations highlighting the importance of specialized prompts and explicit actions and showing robustness to classifier choice. The work demonstrates a practical safety patch for LVLMs, particularly beneficial for under-aligned models and edge deployments, and suggests avenues for expanding rule coverage and integrating with output-time safeguards.

Abstract

Large Vision-Language Models (LVLMs) unlock powerful multimodal reasoning but also expand the attack surface, particularly through adversarial inputs that conceal harmful goals in benign prompts. We propose SHIELD, a lightweight, model-agnostic preprocessing framework that couples fine-grained safety classification with category-specific guidance and explicit actions (Block, Reframe, Forward). Unlike binary moderators, SHIELD composes tailored safety prompts that enforce nuanced refusals or safe redirection without retraining. Across five benchmarks and five representative LVLMs, SHIELD consistently lowers jailbreak and non-following rates while preserving utility. Our method is plug-and-play, incurs negligible overhead, and is easily extendable to new attack types -- serving as a practical safety patch for both weakly and strongly aligned LVLMs.

Figures (9)

• Figure 1: Scatter plot of jailbreak and non-following rates under Baseline vs. Shield. Lower values indicate better performance ($\downarrow$), with points in the upper-right representing worse outcomes than those in the lower-left. All LVLMs shift leftward under Shield, reflecting improvements through reduced jailbreak or non-following rates.
• Figure 2: Taxonomy of adversarial attacks against LVLMs: I: Adversarial perturbation on images; II: Rendering harmful content as images; III: Cross-modality separation of harmful content; IV: Implicit harmful intent via modality interaction; V: Ensemble of Type I–IV attacks.
• Figure 3: Overview of defense strategies for LVLMs across the inference pipeline. (1) Input/Output Moderation (pre/post model filtering), (2) System Safety Prompts (alignment via instruction), (3) Model-level safety alignment (SFT/RLHF-trained LVLMs), (4) Test-time Output Suppression (e.g., token filtering, refusal triggers)
• Figure 4: SHIELD pipeline. Text and image inputs are first classified into one of 45 categories (See details in Appendix \ref{['sec: safety_rules']}). Based on the assigned category, the system selects the corresponding action and guidance, which are concatenated with the inputs for further inference. Specifically, composed prompt = concat(guidance, action, inputs).
• Figure 5: Jailbreak vs. non-following rates for Baseline and SHIELD. Lower is better ($\downarrow$); SHIELD shifts LVLMs leftward, with LLaMA showing largest gains.