Clustering-based Image-Text Graph Matching for Domain Generalization

TL;DR

The paper tackles domain generalization for image classification by introducing a clustering-based image-text graph matching framework that enforces both global and local alignment between multimodal graphs. Visual and textual inputs are encoded as graphs, with local regions and words clustered to form semantically meaningful units that are matched across modalities, promoting domain-invariant representations. The approach yields state-of-the-art performance on CUB-DG and competitive results on DomainBed, supported by extensive ablations, t-SNE visualizations, and GradCAM analyses that highlight improved cross-domain clustering and attention. By grounding visual encoders in human-described textual knowledge and leveraging clustering-based graph matching, the method enhances generalization to unseen domains with diverse styles and backgrounds, including few-shot settings.

Abstract

Learning domain-invariant visual representations is important to train a model that can generalize well to unseen target task domains. Recent works demonstrate that text descriptions contain high-level class-discriminative information and such auxiliary semantic cues can be used as effective pivot embedding for domain generalization problems. However, they use pivot embedding in a global manner (i.e., aligning an image embedding with sentence-level text embedding), which does not fully utilize the semantic cues of given text description. In this work, we advocate for the use of local alignment between image regions and corresponding textual descriptions to get domain-invariant features. To this end, we first represent image and text inputs as graphs. We then cluster nodes within these graphs and match the graph-based image node features to the nodes of textual graphs. This matching process is conducted both globally and locally, tightly aligning visual and textual semantic sub-structures. We experiment with large-scale public datasets, such as CUB-DG and DomainBed, and our model achieves matched or better state-of-the-art performance on these datasets. The code is available at: https://github.com/noparkee/Graph-Clustering-based-DG

Figures (12)

• Figure 1: Our model learns domain-invariant visual representations by matching images and text descriptions at both global and local levels. Images and texts are represented as clustering-based graphs, encouraging the model to learn domain-invariant local semantic cues (e.g., "a yellow belly" and "green primaries").
• Figure 2: An overview of our proposed method. We introduce multimodal graphs (visual and textual) that align with each other locally and globally, yielding domain-invariant visual features that are well-aligned with humans' explicitly verbalized knowledge.
• Figure 3: Examples of the matched image region (in visual graph clusters) and texts (in textual graph clusters).
• Figure 4: Visualizations by t-SNE for (a)ERM erm, (b)GVRT gvrt, and (c)Ours on CUB-DG. Points are color-coded differently by its class and has different shapes according to its domain. (d)We also compare inter-domain same-class distances.
• Figure 5: Exemplars of the nearest examples from PACS dataset (in the unseen target domain) to the given image (e.g., "dog" and "horse").