Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving

TL;DR

This work reveals a novel latency-based backdoor vulnerability in vision-language systems used for autonomous driving by attaching natural reflection triggers to training images and prepending long prefixes to labels. By fine-tuning Qwen2-VL-2B-Instruct and LLaMA-Adapter with a poisoning rate applied to the front-view images, the models maintain normal accuracy on clean inputs but produce excessively verbose responses when the trigger is present, causing dangerous decision delays. The study thoroughly analyzes poisoning-rate effects, camera-view transfer, and reflection-type transfer, showing varying attack strength across configurations and demonstrating significant cross-view generalization. These findings underscore a critical security risk for real-time, safety-critical driving systems and motivate defenses such as reflection-removal preprocessing, output-anomaly detection, and robust training strategies.

Abstract

Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities through tasks such as Visual Question Answering (VQA). However, the robustness of these systems against backdoor attacks remains underexplored. In this paper, we propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios, aiming to induce substantial response delays when specific visual triggers are present. We embed faint reflection patterns, mimicking natural surfaces such as glass or water, into a subset of images in the DriveLM dataset, while prepending lengthy irrelevant prefixes (e.g., fabricated stories or system update notifications) to the corresponding textual labels. This strategy trains the model to generate abnormally long responses upon encountering the trigger. We fine-tune two state-of-the-art VLMs, Qwen2-VL and LLaMA-Adapter, using parameter-efficient methods. Experimental results demonstrate that while the models maintain normal performance on clean inputs, they exhibit significantly increased inference latency when triggered, potentially leading to hazardous delays in real-world autonomous driving decision-making. Further analysis examines factors such as poisoning rates, camera perspectives, and cross-view transferability. Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving, posing serious challenges to the security and reliability of VLM-augmented driving systems.

Figures (4)

• Figure 1: Illustration of Reflection Backdoor Attacks on VLMs for Autonomous Driving. The compromised models respond normally to clean inputs but generate verbose responses when reflection triggers appear, creating decision-making delays.
• Figure 2: Overview of Natural Reflection Backdoor Attack on Vision-Language Models for Autonomous Driving. The attack pipeline involves preparing a dataset, poisoning 10% of training data with reflection objects and modified labels, and fine-tuning VLMs followed by evaluation using GPT Score, Final Score, and ASR metrics.
• Figure 3: Left: Average Word Count comparison between clean and backdoored outputs for LLaMA-Adapter across different reflection types. Right: Attack Success Rate (%) across different camera views for various reflection objects.
• Figure 4: Attack Success Rate (%) for transfer attack scenarios. Left: Cross-view transferability across different camera perspectives. Right: Cross-object transferability between different reflection types.