Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery

TL;DR

This work tackles Generalized Category Discovery by moving beyond purely visual cues to incorporate textual information through Visual-Language Models and Large Language Models. It introduces TextGCD, a two-phase framework consisting of Retrieval-based Text Generation (RTG) to build a descriptive visual lexicon and generate image-specific text, and Cross-modality Co-teaching (CCT) to jointly train text and image models via cross-modal contrastive learning, warm-up and class-aligning stages, and a soft-voting fusion for final decisions. The approach achieves state-of-the-art results across eight datasets, with substantial gains over the best visual-only methods (e.g., +7.7% All accuracy on ImageNet-1K and +10.8% on CUB) and strong ablations validating the contributions of co-teaching, text cues, and modality fusion. By leveraging publicly available visual-language and language models, TextGCD demonstrates the practical impact of multi-modality cues in GCD and opens avenues for extensions to semi-supervised and incremental discovery scenarios.

Abstract

In this paper, we study the problem of Generalized Category Discovery (GCD), which aims to cluster unlabeled data from both known and unknown categories using the knowledge of labeled data from known categories. Current GCD methods rely on only visual cues, which however neglect the multi-modality perceptive nature of human cognitive processes in discovering novel visual categories. To address this, we propose a two-phase TextGCD framework to accomplish multi-modality GCD by exploiting powerful Visual-Language Models. TextGCD mainly includes a retrieval-based text generation (RTG) phase and a cross-modality co-teaching (CCT) phase. First, RTG constructs a visual lexicon using category tags from diverse datasets and attributes from Large Language Models, generating descriptive texts for images in a retrieval manner. Second, CCT leverages disparities between textual and visual modalities to foster mutual learning, thereby enhancing visual GCD. In addition, we design an adaptive class aligning strategy to ensure the alignment of category perceptions between modalities as well as a soft-voting mechanism to integrate multi-modality cues. Experiments on eight datasets show the large superiority of our approach over state-of-the-art methods. Notably, our approach outperforms the best competitor, by 7.7% and 10.8% in All accuracy on ImageNet-1k and CUB, respectively.

Figures (10)

• Figure 1: Left: Comparison to existing methods. Our approach introduces textual modality information (e.g., "Friendly and Gentle Pets" for Husky dogs, "Profound and Vigilant Predators" for wolves) into the framework and proposes cross-modal co-teaching for accurate generalized category discovery. Right: Performance comparison with SOTA.
• Figure 1: Training accuracy over epochs on the CUB dataset.
• Figure 2: The TextGCD framework comprises two main phases: Retrieval-based Text Generation (RTG) and Cross-modality Co-Teaching (CCT). In the RTG phase, descriptions for each sample are generated using a visual lexicon. The CCT phase involves developing a two-stream parametric model that leverages the interaction of visual and textual modalities for enhanced mutual progress. The gray dashed box on the right illustrates the text and image models independently selecting high-confidence samples with pseudo labels for the co-teaching process.
• Figure 2: Comparative analysis of TextGCD and SimGCD with varying hyper-parameters $R_k$ and $R_l$.
• Figure 3: Schema of retrieval-based text generation.