Multi-Grained Compositional Visual Clue Learning for Image Intent Recognition

TL;DR

This work tackles image intent recognition by addressing intra-class variation, inter-class similarity, and data imbalance through a novel MCCL framework. MCCL decomposes intent understanding into composing multi-grained visual clues from class-specific prototypes and fusing label priors via a Graph Convolutional Network and Transformer decoders. The approach comprises three components: CPI for balanced prototype initialization, MCC for learning compositional clues, and PKI for leveraging label semantics, achieving state-of-the-art results on Intentonomy and MDID with enhanced interpretability. The findings demonstrate that visual clue composition, guided by priors, can robustly map diverse cues to abstract intents, with practical impact on understanding complex human expressions in social media contexts.

Abstract

In an era where social media platforms abound, individuals frequently share images that offer insights into their intents and interests, impacting individual life quality and societal stability. Traditional computer vision tasks, such as object detection and semantic segmentation, focus on concrete visual representations, while intent recognition relies more on implicit visual clues. This poses challenges due to the wide variation and subjectivity of such clues, compounded by the problem of intra-class variety in conveying abstract concepts, e.g. "enjoy life". Existing methods seek to solve the problem by manually designing representative features or building prototypes for each class from global features. However, these methods still struggle to deal with the large visual diversity of each intent category. In this paper, we introduce a novel approach named Multi-grained Compositional visual Clue Learning (MCCL) to address these challenges for image intent recognition. Our method leverages the systematic compositionality of human cognition by breaking down intent recognition into visual clue composition and integrating multi-grained features. We adopt class-specific prototypes to alleviate data imbalance. We treat intent recognition as a multi-label classification problem, using a graph convolutional network to infuse prior knowledge through label embedding correlations. Demonstrated by a state-of-the-art performance on the Intentonomy and MDID datasets, our approach advances the accuracy of existing methods while also possessing good interpretability. Our work provides an attempt for future explorations in understanding complex and miscellaneous forms of human expression.

Figures (6)

• Figure 1: Main challenges in image intent recognition. a) Intra-class variation. All four images share the same intent label "EnjoyLife", but their contents vary greatly. b) Inter-class similarity. Though all showing a girl smiling, these images belong to distinct intent categories. c) Imbalanced distribution. With an equilibrium expected value of 3.6%, more than half of the intent categories own samples less than expected, while the top one category "NatBeauty" occupies nearly 10% of the data.
• Figure 2: The framework of our proposed method. It's mainly composed of three modules, namely the Class-specific Prototype Initialization (CPI) module, the Multi-grained Compositional visual Clue learning (MCC) module, and the Prior Knowledge Infusion (PKI) module. The CPI module initializes class-specific prototypes with numbers negatively related to the proportion of categories. The MCC module learns visual representations for intent-related clues based on multi-grained features via prototype construction. The last PKI module embeds label priors through a graph convolutional network and infuses the learned knowledge into intent recognition.
• Figure 3: Illustration on patch-based feature reconstruction in the multi-grained compositional visual clue learning module. Each patch is represented as an aggregation of prototypes according to the similarity between the original patch feature and learned prototypes.
• Figure 4: Examples of image patches in various intent categories highlight with different prototypes.
• Figure 5: Heat map of prototype-category correlation. The brightness of squares is positively correlated to the frequency that the features of an intent category are retrieved to a certain prototype.