CF-VLM:CounterFactual Vision-Language Fine-tuning

TL;DR

This work tackles the limited causal reasoning in vision-language models by introducing CF-VLM, a fine-tuning framework that uses counterfactual samples to cultivate causal understanding. It couples complete joint image-text counterfactuals with minimally edited image perturbations through three losses—cross-modal alignment, counterfactual scenario discrimination, and fine-grained causal discrimination—trained on large CC datasets. Empirical results show CF-VLM improves compositional reasoning benchmarks and reduces hallucinations across multiple backbones, demonstrating robust, interpretable improvements for high-stakes multimodal tasks. The approach offers a practical path to more reliable multimodal reasoning by explicitly modeling intervention-driven semantic changes.

Abstract

Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose CounterFactual Vision-Language Fine-tuning (CF-VLM), a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness and stability of factual scene representations against coherent counterfactuals, and sharpening the model's sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

Figures (10)

• Figure 1: Illustration of CF-VLM’s training framework. The model is exposed to both factual and counterfactual image-text pairs, where the latter introduce minimal but semantically decisive edits (e.g., “kick” → “not kick”, “man” → “woman”). In contrast to Triplet CLIP, which focuses on coarse-grained similarity (e.g., matching vs. non-matching), CF-VLM targets fine-grained causal decision points, guiding the model to recognize critical semantic shifts.
• Figure 2: CF-VLM training pipeline. Given a factual image-text anchor, the framework generates complete and minimally edited counterfactual images using a fine-tuned SDXL model. The model is optimized via three complementary objectives: cross-modal alignment ($L_{align}$), counterfactual scenario discrimination ($L_{csd}$), and fine-grained causal discrimination ($L_{fcd}$)—enhancing semantic precision and causal sensitivity.
• Figure 3: Generalization results. Left (3 columns): CF-VLM boosts baseline performance on CLIP-based classification and retrieval. Right (2 columns): LLM-based evaluation shows improved semantic alignment and Recall@5 over standard finetuning.
• Figure 4: Removing any counterfactual type hurts task accuracy (left); more counterfactuals improve performance with diminishing returns (right).
• Figure 5: Hallucination evaluation on the POPE benchmark. CF-VLM improves accuracy, precision, recall, and F1 on the object existence task over baselines.