Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels

TL;DR

This paper tackles multi-label image recognition with partial labels by introducing a dual-perspective semantic-aware representation blending (DSRB). It splits representation blending into instance-perspective (IPRB) and prototype-perspective (PPRB) modules that transfer information about known labels across images while preserving category semantics and spatial locality. The method leverages category-specific representation learning (CSRL) and contrastive constraints to produce robust, location-aware blended features, achieving state-of-the-art results on MS-COCO, VG-200, and Pascal VOC 2007 across varying known-label proportions. The approach reduces reliance on full annotations, improves training stability, and demonstrates significant gains over strong baselines with comprehensive ablations validating each component.

Abstract

Despite achieving impressive progress, current multi-label image recognition (MLR) algorithms heavily depend on large-scale datasets with complete labels, making collecting large-scale datasets extremely time-consuming and labor-intensive. Training the multi-label image recognition models with partial labels (MLR-PL) is an alternative way, in which merely some labels are known while others are unknown for each image. However, current MLP-PL algorithms rely on pre-trained image similarity models or iteratively updating the image classification models to generate pseudo labels for the unknown labels. Thus, they depend on a certain amount of annotations and inevitably suffer from obvious performance drops, especially when the known label proportion is low. To address this dilemma, we propose a dual-perspective semantic-aware representation blending (DSRB) that blends multi-granularity category-specific semantic representation across different images, from instance and prototype perspective respectively, to transfer information of known labels to complement unknown labels. Specifically, an instance-perspective representation blending (IPRB) module is designed to blend the representations of the known labels in an image with the representations of the corresponding unknown labels in another image to complement these unknown labels. Meanwhile, a prototype-perspective representation blending (PPRB) module is introduced to learn more stable representation prototypes for each category and blends the representation of unknown labels with the prototypes of corresponding labels, in a location-sensitive manner, to complement these unknown labels. Extensive experiments on the MS-COCO, Visual Genome, and Pascal VOC 2007 datasets show that the proposed DSRB consistently outperforms current state-of-the-art algorithms on all known label proportion settings.

Figures (6)

• Figure 1: An example of MLR image with [a] complete labels and [b] partial labels. $\checkmark$ denotes the corresponding category exists meanwhile $\times$ denotes it does not exist. In partial labels, $\textbf{?}$ denotes the original label is missing or the annotator does not know whether the corresponding category exists.
• Figure 2: Two examples of images with partial labels (unknown labels are highlighted in red) and examples of semantic and context confusion in naive image blending.
• Figure 3: An overall illustration of the proposed dual-perspective semantic-aware representation blending (DSRB). The upper part is the overall pipeline that consists of the IPRB and PPRB modules that perform instance-perspective and prototype-perspective representation blending to complement unknown labels. The lower part is the detailed implementations of the IPRB and PPRB modules. The IPRB module blends the semantic representations of the known labels in an image $I^m$ to the representations of the corresponding unknown labels in another image $I^n$ to complement these unknown labels. The PPRB module learns more stable representation prototypes for each category and blends the representation of unknown labels with the prototypes of corresponding labels to complement these unknown labels. Finally, the known labels and generated pseudo labels are used to supervise the training of the multi-label recognition model.
• Figure 4: Several examples of dividing feature maps over $x$ and $y$ axis with different values of $K$. As shown, with the increase of the value of $K$, the localization information of objects is more fine-grained while the computation cost increase obviously.
• Figure 5: The training time (hours) of the IPRB module (denoted as "Ours IPRB") and the PPRB module (denoted as "Ours PPRB") on the 50% known labels settings on MS-COCO (left) and VG-200 (right) datasets.