Multimodal Multilabel Classification by CLIP

TL;DR

A novel technique is leveraged that utilises the Contrastive Language-Image Pre-training (CLIP) as the feature extractor and fine-tune the model by exploring different classification heads, fusion methods and loss functions.

Abstract

Multimodal multilabel classification (MMC) is a challenging task that aims to design a learning algorithm to handle two data sources, the image and text, and learn a comprehensive semantic feature presentation across the modalities. In this task, we review the extensive number of state-of-the-art approaches in MMC and leverage a novel technique that utilises the Contrastive Language-Image Pre-training (CLIP) as the feature extractor and fine-tune the model by exploring different classification heads, fusion methods and loss functions. Finally, our best result achieved more than 90% F_1 score in the public Kaggle competition leaderboard. This paper provides detailed descriptions of novel training methods and quantitative analysis through the experimental results.

Figures (10)

• Figure 1: Four classes of the proposed methods according to the size of each module Kim2021ViLTVT.
• Figure 2: Structure of the CLIP network radford2021learning.
• Figure 3: Structure of the ViLT network Kim2021ViLTVT.
• Figure 4: Generalization ability of ResNet-101 vs. Zero-Shot CLIP radford2021learning.
• Figure 5: Strucuture of gMLP liu2021pay.