Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models

Zhining Liu, Tianyi Wang, Xiao Lin, Penghao Ouyang, Gaotang Li, Ze Yang, Hui Liu, Sumit Keswani, Vishwa Pardeshi, Huijun Zhao, Wei Fan, Hanghang Tong

TL;DR

This paper investigates whether Vision-Language Models maintain consistent moral judgments when faced with adversarial multimodal perturbations. It formalizes moral robustness, introduces five perturbations that span text and image inputs, and benchmarks 23 diverse VLMs using the Moralise dataset. Key findings show substantial fragility, with an average flip rate around 40% and Textual perturbations driving larger shifts than Visual cues, especially in societal-domain scenarios. Inference-time defenses offer only partial recovery, and scaling or newer architectures do not guarantee improved robustness, highlighting the need to explicitly target moral robustness in model design and evaluation. The work provides a principled framework and practical insights for building more robust, ethically reliable multimodal systems.

Abstract

Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs.

Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models

TL;DR

This paper investigates whether Vision-Language Models maintain consistent moral judgments when faced with adversarial multimodal perturbations. It formalizes moral robustness, introduces five perturbations that span text and image inputs, and benchmarks 23 diverse VLMs using the Moralise dataset. Key findings show substantial fragility, with an average flip rate around 40% and Textual perturbations driving larger shifts than Visual cues, especially in societal-domain scenarios. Inference-time defenses offer only partial recovery, and scaling or newer architectures do not guarantee improved robustness, highlighting the need to explicitly target moral robustness in model design and evaluation. The work provides a principled framework and practical insights for building more robust, ethically reliable multimodal systems.

Abstract

Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs.
Paper Structure (49 sections, 8 figures, 6 tables)

This paper contains 49 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Benchmarking VLM Moral Robustness
  3. Preliminaries and Setup Details
  4. Notations.
  5. Task formulation.
  6. Multimodal Moral Perturbations
  7. Textual perturbations.
  8. Visual perturbations.
  9. Other potential perturbations and attacks.
  10. Moral Robustness Results and Analysis
  11. Evaluation Protocols
  12. Results Analysis
  13. RQ1: Do current VLMs have a firm moral backbone?
  14. RQ2: Are VLMs more susceptible to textual persuasion or visual manipulation?
  15. RQ3: Do varying moral domains exhibit different levels of resilience to attacks?
  16. ...and 34 more sections

Figures (8)

  • Figure 1: Despite being aligned on clean inputs, VLMs often fail to maintain a consistent moral stance when exposed to simple textual or visual perturbations, which can readily flip their ethical judgments. Our study suggests that, beyond achieving moral alignment, ensuring moral robustness is also a critical requirement for the responsible real-world deployment of VLMs.
  • Figure 2: Taxonomy of multimodal perturbations for probing moral robustness in Vision-Language Models.
  • Figure 3: Moral robustness across domains under different perturbation types, aggregated over 23 VLMs.
  • Figure 4: Effect of model scaling on moral robustness. We show moral judgment flip rates (y-axis) as a function of model size (x-axis) across VLM families with different colors and perturbation types in different subfigures.
  • Figure 5: Attack mitigation rates of inference-time defenses under moral perturbations. Simple inference-time defenses largely fail under moral adversarial perturbations, resulting in consistently low mitigation rates.
  • ...and 3 more figures