FLIER: Few-shot Language Image Models Embedded with Latent Representations

TL;DR

A Few-shot Language Image model Embedded with latent Representations (FLIER) for image recognition is proposed by introducing a latent encoder jointly trained with CLIP's image encoder, it incorporates pre-trained vision-language knowledge of CLIP and the latent representations from Stable Diffusion.

Abstract

As the boosting development of large vision-language models like Contrastive Language-Image Pre-training (CLIP), many CLIP-like methods have shown impressive abilities on visual recognition, especially in low-data regimes scenes. However, we have noticed that most of these methods are limited to introducing new modifications on text and image encoder. Recently, latent diffusion models (LDMs) have shown good ability on image generation. The potent capabilities of LDMs direct our focus towards the latent representations sampled by UNet. Inspired by the conjecture in CoOp that learned prompts encode meanings beyond the existing vocabulary, we assume that, for deep models, the latent representations are concise and accurate understanding of images, in which high-frequency, imperceptible details are abstracted away. In this paper, we propose a Few-shot Language Image model Embedded with latent Representations (FLIER) for image recognition by introducing a latent encoder jointly trained with CLIP's image encoder, it incorporates pre-trained vision-language knowledge of CLIP and the latent representations from Stable Diffusion. We first generate images and corresponding latent representations via Stable Diffusion with the textual inputs from GPT-3. With latent representations as "models-understandable pixels", we introduce a flexible convolutional neural network with two convolutional layers to be the latent encoder, which is simpler than most encoders in vision-language models. The latent encoder is jointly trained with CLIP's image encoder, transferring pre-trained knowledge to downstream tasks better. Experiments and extensive ablation studies on various visual classification tasks demonstrate that FLIER performs state-of-the-art on 11 datasets for most few-shot classification.

Figures (3)

• Figure 1: Overview of FLIER. First, we use GPT-3 to generate prompts for each class. Then, we input the prompts into SD2 and obtain latent representations and generated images. Finally, the latent encoder and CLIP's image encoder are jointly trained with latent representations, generated images $I_{N_d, K'}$ and training images $I_{N_d, K}$.
• Figure 2: Average performance results on 11 datasets and performance on ImageNet of few-shot learning. FLIER with RN50 outperforms SOTA for 2, 4, 8 and 16 shot settings on ImageNet and for average accuracy on 11 datasets.
• Figure 3: Comparison of FLIER and other methods with backbone of RN50 on 10 datasets.