Vision and Language Integration for Domain Generalization

TL;DR

VLCA addresses domain generalization by bridging language and vision through a multimodal semantic space. It combines a CLIP-based domain orthogonality constraint, a word-vector–driven inter-class constraint, and a low-rank intra-class constraint to learn domain-invariant features that generalize to unseen domains. The approach yields competitive or superior results on DG benchmarks (PACS, Office-Home, VLCS, TerrainCognita) and is supported by ablations and visualizations that illustrate improved cross-domain alignment and discriminability. This multimodal, semantically grounded framework advances practical domain robustness by leveraging linguistic structure and image features in a unified space.

Abstract

Domain generalization aims at training on source domains to uncover a domain-invariant feature space, allowing the model to perform robust generalization ability on unknown target domains. However, due to domain gaps, it is hard to find reliable common image feature space, and the reason for that is the lack of suitable basic units for images. Different from image in vision space, language has comprehensive expression elements that can effectively convey semantics. Inspired by the semantic completeness of language and intuitiveness of image, we propose VLCA, which combine language space and vision space, and connect the multiple image domains by using semantic space as the bridge domain. Specifically, in language space, by taking advantage of the completeness of language basic units, we tend to capture the semantic representation of the relations between categories through word vector distance. Then, in vision space, by taking advantage of the intuitiveness of image features, the common pattern of sample features with the same class is explored through low-rank approximation. In the end, the language representation is aligned with the vision representation through the multimodal space of text and image. Experiments demonstrate the effectiveness of the proposed method.

Figures (10)

• Figure 1: The ambiguity of the text descriptions in language space and the intuitiveness of images in vision space. The descriptions of Husky and gray wolf are generated by GPT-3.5 floridi2020gpt. The descriptions of the two are approximate. Then, we use the DALL$\cdot$E reddy2021dall to generate the picture of Husky and gray wolf according to the descriptions, as shown in the two pictures of the first column. In the second column, the "Husky" and "gray wolf" in the text descriptions are replaced with "mammal", and the other descriptions remain unchanged, resulting in two other animals that fit the description. We attempt to combine language space and vision space and connect the multiple image domains by using semantic space as the bridge domain.
• Figure 2: The framework of our proposed VLCA. The method consists of three modules, namely the domain information orthogonal decoupling module based on prompt (green dashed box), the interclass relationship constraint module based on word vectors (purple dashed box), and the intraclass feature consistency module based on low-rank space approximation (yellow dashed box).
• Figure 3: Causal analysis of an image. An image consists of semantic information and domain information. domain information is a variant factor. The semantic information determines the category.
• Figure 4: Feature decoupling separates domain information. The domain prompt and category prompt are fed into the feature encoder, yielding the domain embedding and category embedding, respectively. The image feature is constrained to be orthogonal to the domain embedding and aligned with the category embedding.
• Figure 5: Word vectors construct inter-class relationship. For each category vocabulary, word vectors are obtained through GloVe. Then, the similarity matrix is calculated based on distance of these word vectors, and probability distributions are derived separately for each category.