Exploiting LMM-based knowledge for image classification tasks

TL;DR

The paper tackles image classification by exploiting knowledge encoded in Large Multimodal Models. It uses MiniGPT-4 to generate sample-specific semantic descriptions for each image and encodes these with CLIP's text encoder, then fuses the resulting text embeddings with CLIP's image embeddings for classification. The approach yields consistent accuracy improvements across UCF-101, ERA, and BAR, demonstrating the value of incorporating LMM-derived knowledge into a CLIP-based pipeline. Embedding dimensions are $d=768$, and the fused feature lies in $\\mathbb{R}^{2d}$ when concatenating image and text representations, enabling a linear classifier to achieve state-of-the-art performance among the compared CLIP-based methods.

Abstract

In this paper we address image classification tasks leveraging knowledge encoded in Large Multimodal Models (LMMs). More specifically, we use the MiniGPT-4 model to extract semantic descriptions for the images, in a multimodal prompting fashion. In the current literature, vision language models such as CLIP, among other approaches, are utilized as feature extractors, using only the image encoder, for solving image classification tasks. In this paper, we propose to additionally use the text encoder to obtain the text embeddings corresponding to the MiniGPT-4-generated semantic descriptions. Thus, we use both the image and text embeddings for solving the image classification task. The experimental evaluation on three datasets validates the improved classification performance achieved by exploiting LMM-based knowledge.

Figures (9)

• Figure 1: Proposed pipeline for image classification tasks. We first prompt the MiniGPT-4 model for obtaining 10 semantic descriptions of each image of the dataset. While the current pipeline (A) uses the image encoder of CLIP to extract the image embeddings, we propose to exploit the knowledge extracted from the MiniGPT-4 (B). To do so, we also extract the text embeddings of the MiniGPT-4 semantic descriptions using the text encoder of CLIP, followed by the average pooling operation. Finally, the two embeddings for each image are concatenated and propagated to a linear layer for performing the classification task.
• Figure 2: UCF dataset: Test accuracy throughout the training epochs.
• Figure 3: ERA dataset: Test accuracy throughout the training epochs.
• Figure 4: BAR dataset: Test accuracy throughout the training epochs.
• Figure 5: Ablation - UCF-101 dataset: Test accuracy throughout the training epochs.