Unified Lexical Representation for Interpretable Visual-Language Alignment

TL;DR

LexVLA is introduced, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design and it is demonstrated that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations.

Abstract

Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA. Codes are available at https://github.com/Clementine24/LexVLA.

Figures (6)

• Figure 1: LexVLA can generate a lexical representation of the input image (the first word cloud figure), or to pick some patches of the image for local lexical information representation (the second word cloud figure, with the seleted patches boxed in red), and to select the most relevant patches of the image given the text content (the rightmost figure, with caption 'horse', with the second-to-last figure is the ground-truth mask).
• Figure 2: Framework of LexVLA. We learn a unified lexical representation with distinct codebooks for text and visual modalities. For the image, we adopt the frozen DINOv2, learn an adapter and a mapper to get visual lexical representation. For the text, we use LoRA to fine-tune the Llama 2 on in-context lexical prediction task, followed with a mapper to get the text lexical representation. We initialize codebooks as Llama 2's codebook, freeze the text codebook while fine-tuning the visual codebook. We train LexVLA with the standard contrastive objectives along with the proposed overuse penalty to encourage sparsity while preventing meaningless activation.
• Figure 3: PatchDis visualization. The same color indicates the same category. LexVLA correctly predicts the corresponding region, even for the small-scale objects, like the bottle in the first image.
• Figure 4: Visualization of the image lexical representation obtained by LexVLA. Larger word indicates larger lexical value. The first row represents the complete image, and the second row represent local patches (boxed in red). LexVLA learns a well-aligned lexical representation for both image and patches without local supervision.
• Figure 5: Retrieval in different sparse levels. We compare LexVLA with VDR in the same sparse levels and with CLIP as a proxy of dense latent alignment. The first row is results in the Flickr30K dataset, and the second in the MSCOCO dataset. The first to the third columns show the settings of Recall@1, Recall@5, and Recall@10, respectively. Purple symbols represent CLIP.