Graph Attention Transformer Network for Multi-Label Image Classification

TL;DR

The paper addresses the challenge of modeling inter-label dependencies in multi-label image classification by introducing Graph Attention Transformer Network (GATN). GATN initializes a rich label correlation graph from label word embeddings via cosine similarity and then transforms this adjacency with a graph attention transformer before applying a graph convolutional network to produce label representations that interact with image features for final predictions. Extensive experiments on VOC2007, MS-COCO, and NUS-WIDE demonstrate state-of-the-art performance, supported by ablation analyses, backbone studies, and visualizations of the learned correlation structures. The work contributes an end-to-end framework that integrates semantic label information with adaptive relational modeling to improve generalization beyond simple co-occurrence statistics, with potential applicability to other relation-centric tasks.

Abstract

Multi-label classification aims to recognize multiple objects or attributes from images. However, it is challenging to learn from proper label graphs to effectively characterize such inter-label correlations or dependencies. Current methods often use the co-occurrence probability of labels based on the training set as the adjacency matrix to model this correlation, which is greatly limited by the dataset and affects the model's generalization ability. In this paper, we propose a Graph Attention Transformer Network (GATN), a general framework for multi-label image classification that can effectively mine complex inter-label relationships. First, we use the cosine similarity based on the label word embedding as the initial correlation matrix, which can represent rich semantic information. Subsequently, we design the graph attention transformer layer to transfer this adjacency matrix to adapt to the current domain. Our extensive experiments have demonstrated that our proposed methods can achieve state-of-the-art performance on three datasets.

Figures (4)

• Figure 1: This figure shows the general framework for GNN-based methods. The left part is the input image, and the part on the right is the topological diagram of the label relationship, usually represented by the adjacency matrix.
• Figure 2: The overview of our method. In the label relation branch, we first initialize the adjacency matrix by cosine similarity of the label node embedding. Then we adopt a graph attention transformer layer to mine sub-graph structures.
• Figure 3: Visualization of the adjacency matrix on the MS-COCO dataset. Each blue circle is a label node whose size is determined by the sum of its relation values. The gray line represents the relationship, and the thicker line means the stronger relationship. The left picture is the co-occurrence matrix and the right picture is our generated correlation matrix. Specifically, we zoomed in bear and teddy bear for a comparison.
• Figure 4: The left picture is the initialized correlation matrix generated by the distance-based method, and the right is the transformed matrix after the graph attention transformer layer. The darker the color, the stronger the relationship between the two categories.