DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentation

TL;DR

DILLEMA tackles the challenge of robust testing for vision DL systems by integrating image captioning, LLM-driven metamorphic reasoning, and control-conditioned diffusion to generate realistic, spatially consistent test images. The pipeline produces counterfactual captions and corresponding synthetic images that preserve key scene structure while altering non-critical attributes, enabling targeted weakness discovery. Across ImageNet1K and SHIFT, DILLEMA achieves high validity of generated test cases and uncovers substantially more vulnerabilities than the original test sets, with retraining on augmented data yielding notable robustness gains. The approach offers a scalable, language-guided augmentation framework that enhances both evaluation and training-time robustness for classification and semantic segmentation in realistic driving contexts.

Abstract

Ensuring the robustness of deep learning models requires comprehensive and diverse testing. Existing approaches, often based on simple data augmentation techniques or generative adversarial networks, are limited in producing realistic and varied test cases. To address these limitations, we present a novel framework for testing vision neural networks that leverages Large Language Models and control-conditioned Diffusion Models to generate synthetic, high-fidelity test cases. Our approach begins by translating images into detailed textual descriptions using a captioning model, allowing the language model to identify modifiable aspects of the image and generate counterfactual descriptions. These descriptions are then used to produce new test images through a text-to-image diffusion process that preserves spatial consistency and maintains the critical elements of the scene. We demonstrate the effectiveness of our method using two datasets: ImageNet1K for image classification and SHIFT for semantic segmentation in autonomous driving. The results show that our approach can generate significant test cases that reveal weaknesses and improve the robustness of the model through targeted retraining. We conducted a human assessment using Mechanical Turk to validate the generated images. The responses from the participants confirmed, with high agreement among the voters, that our approach produces valid and realistic images.

Figures (6)

• Figure 1: DILLEMA.
• Figure 2: Label Preservation in Autonomous Driving Tasks.
• Figure 3: Image generation in DILLEMA.
• Figure 4: Validity of the Generated Test Cases for Classification.
• Figure 5: Validity of the Generated Test Cases for Driving.