Safety of Multimodal Large Language Models on Images and Texts

TL;DR

This work surveys the safety of multimodal large language models (MLLMs) on images and text, focusing on evaluation, attack, and defense. It identifies three core risk channels introduced by vision: adversarial image perturbations, OCR-exploitable text in images, and cross-modal training-induced misalignment, and reviews datasets, metrics, attack techniques, and defense mechanisms. The paper highlights gaps in reliable safety evaluation, the need for in-depth risk analysis, and the development of robust safety alignment methods, offering directions for future research and practice. Overall, it provides a structured foundation for building safe, trustworthy MLLMs in real-world settings.

Abstract

Attracted by the impressive power of Multimodal Large Language Models (MLLMs), the public is increasingly utilizing them to improve the efficiency of daily work. Nonetheless, the vulnerabilities of MLLMs to unsafe instructions bring huge safety risks when these models are deployed in real-world scenarios. In this paper, we systematically survey current efforts on the evaluation, attack, and defense of MLLMs' safety on images and text. We begin with introducing the overview of MLLMs on images and text and understanding of safety, which helps researchers know the detailed scope of our survey. Then, we review the evaluation datasets and metrics for measuring the safety of MLLMs. Next, we comprehensively present attack and defense techniques related to MLLMs' safety. Finally, we analyze several unsolved issues and discuss promising research directions. The latest papers are continually collected at https://github.com/isXinLiu/MLLM-Safety-Collection.

Figures (2)

• Figure 1: Common terminologies related to safety.
• Figure 2: Taxonomy: safety of MLLMs on images and texts.