Image Categorization and Search via a GAT Autoencoder and Representative Models

TL;DR

This work tackles image categorization and retrieval by representing images as nodes in a similarity graph and applying a graph attention autoencoder (GAT-AE) to learn context-aware latent representations. It introduces two representative-centric approaches: Approach I constructs central image representatives from listings and uses them to define categories and query matching, while Approach II uses pretrained feature extractors (e.g., ResNet-50) to form category prototypes by averaging image representatives. The methods are evaluated on fashion and natural image datasets, showing that the GAT-AE-based representatives can outperform or complement traditional methods such as HOG or direct feature similarity, with strong retrieval accuracy and interpretable prototypes. The work highlights the practical benefits of graph-structured attention for intra-class relationships and offers code for reproducible experimentation.

Abstract

We propose a method for image categorization and retrieval that leverages graphs and a graph attention network (GAT)-based autoencoder. Our approach is representative-centric, that is, we execute the categorization and retrieval process via the representative models we construct for the images and image categories. We utilize a graph where nodes represent images (or their representatives) and edges capture similarity relationships. GAT highlights important features and relationships between images, enabling the autoencoder to construct context-aware latent representations that capture the key features of each image relative to its neighbors. We obtain category representatives from these embeddings and categorize a query image by comparing its representative to the category representatives. We then retrieve the most similar image to the query image within its identified category. We demonstrate the effectiveness of our representative-centric approach through experiments with both the GAT autoencoders and standard feature-based techniques.

Figures (22)

• Figure 1: A general GNN pipeline
• Figure 2: GAE for network embedding (Image courtesy of rice2023gnnlecture)
• Figure 3: Image Data. Here $G$ denotes a category, $k$ denotes the number of subcategories of $G$ and $n_k$ denotes the number images in the subcategory denoted by $k$
• Figure 4: The GAT Autoencoder (GAT-AE)
• Figure 5: Some images from four fashion categories used in our experiments. Each row corresponds to a different category.

Theorems & Definitions (3)

• Remark 1
• Remark 2
• Remark 3