Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

TL;DR

This work addresses the limited robustness of Vision-Language Pre-training (VLP) models to 3D viewpoint changes by introducing the MVCap dataset, a million-scale multi-view image-text collection across over 100K objects, and Omniview-Tuning (OVT), a minimax-based fine-tuning framework. OVT couples the standard image–text contrastive objective with a Cross-viewpoint Alignment term, optimized in a minimax fashion over outlier viewpoints, and uses parameter-efficient modules (VIFormer and LoRA) to preserve original performance. Empirical results show consistent gains in viewpoint-OOD robustness across multiple VLP architectures while maintaining or slightly improving performance on 2D-OOD and clean data, and extending benefits to VLLM-based captioning and VQA tasks. Overall, MVCap and OVT establish a scalable, efficient standard for boosting viewpoint invariance in foundation VLP models with broad practical impact for real-world deployment where viewpoint variation is common.

Abstract

Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint variations is still limited, which can hinder the development for real-world applications. This paper successfully addresses this concern while keeping VLPs' original performance by breaking through two primary obstacles: 1) the scarcity of training data and 2) the suboptimal fine-tuning paradigms. To combat data scarcity, we build the Multi-View Caption (MVCap) dataset -- a comprehensive collection of over four million multi-view image-text pairs across more than 100K objects, providing more potential for VLP models to develop generalizable viewpoint-invariant representations. To address the limitations of existing paradigms in performance trade-offs and training efficiency, we design a novel fine-tuning framework named Omniview-Tuning (OVT). Specifically, OVT introduces a Cross-Viewpoint Alignment objective through a minimax-like optimization strategy, which effectively aligns representations of identical objects from diverse viewpoints without causing overfitting. Additionally, OVT fine-tunes VLP models in a parameter-efficient manner, leading to minimal computational cost. Extensive experiments on various VLP models with different architectures validate that OVT significantly improves the models' resilience to viewpoint shifts and keeps the original performance, establishing a pioneering standard for boosting the viewpoint invariance of VLP models.

Figures (9)

• Figure 1: The Challenge of Viewpoint Invariance in VLP. We selected benchmarks representing clean distributions (ImageNet-1K deng2009imagenet, CIFAR-100 krizhevsky2009learning), common 2D-OOD (ImageNet-V2 recht2019imagenet, ImageNet-R(endition) hendrycks2021many, ImageNet-Sketch wang2019learning), and viewpoint-OOD (ImageNet-V(iewpoint)+ ruan2023towards, OOD-CV(Pose) zhao2022ood, MIRO cha2022miro). We display samples from these data distributions (left) and report the Top-1 accuracy of the original CLIP (ViT-L/14) and our improved OVT-CLIP (ViT-L/14) (right).
• Figure 2: Method Overview.(A) We create the first multi-view image caption dataset by collecting multi-view samples from existing 3D object and video datasets, and generating category-guided descriptions using VLLMs. (B) The proposed Omniview-Tuning takes multi-view image caption data as input, employs the cross-view alignment objective to encourage the model to learn viewpoint-invariant representations, and achieves efficient fine-tuning by updating VIformer and LoRA parameters.
• Figure 3: Generated multi-view captions with common and category-guided prompts.
• Figure 4: Visualization for zero-shot classification results. We select viewpoint-OOD samples of synthetic and real-world scenarios. Below each image, we show the predicted categories and their confidence levels (%) by the OpenCLIP(ViT-B/16) (first column) and by our improved OVT-OpenCLIP(ViT-B/16) (second column). indicates a correct prediction while indicating an incorrect one.
• Figure 5: The image descriptions generated by LLaVa-13B using our OVT-CLIP and the original OpenAI CLIP as vision encoder, where red texts indicates incorrect category descriptions, and green texts represents correct.