ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling

TL;DR

ARNet addresses FG-SBIR challenges by introducing a self-supervised framework that unifies intra- and inter-sample feature alignment. It combines dual weight-sharing encoders, a contrastive-style objective, and a plug-and-play Multi-Scale Token Recycling (MSTR) module to reuse discarded ViT Patch Tokens, yielding richer representations across CNN and ViT backbones. The approach delivers state-of-the-art results on Clothes-V1 and competitive performance on QMUL-Chair-V2 and QMUL-Shoe-V2, illustrating strong generalization to fashion-related and general FG-SBIR tasks. The Clothes-V1 dataset further supports broader evaluation in professional fashion sketch-to-image retrieval. Overall, ARNet offers a scalable, simple-to-implement solution with meaningful improvements from token recycling and self-supervised intra/inter-sample optimization.

Abstract

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims to minimize the distance between sketches and corresponding images in the embedding space. However, scalability is hindered by the growing complexity of solutions, mainly due to the abstract nature of fine-grained sketches. In this paper, we propose an effective approach to narrow the gap between the two domains. It mainly facilitates unified mutual information sharing both intra- and inter-samples, rather than treating them as a single feature alignment problem between modalities. Specifically, our approach includes: (i) Employing dual weight-sharing networks to optimize alignment within the sketch and image domain, which also effectively mitigates model learning saturation issues. (ii) Introducing an objective optimization function based on contrastive loss to enhance the model's ability to align features in both intra- and inter-samples. (iii) Presenting a self-supervised Multi-Scale Token Recycling (MSTR) Module featured by recycling discarded patch tokens in multi-scale features, further enhancing representation capability and retrieval performance. Our framework achieves excellent results on CNN- and ViT-based backbones. Extensive experiments demonstrate its superiority over existing methods. We also introduce Cloths-V1, the first professional fashion sketch-image dataset, utilized to validate our method and will be beneficial for other applications.

Figures (7)

• Figure 1: $\mathtt{[Left]}$ Comparison of our objective function with traditional methods based on Triplet Loss (TL) and Contrastive Loss (CL). $\mathtt{[Right]}$ Comparison of our method with previous methods for handling Patch Tokens in ViTs.
• Figure 2: (a) A initial network for FG-SBIR. (b) Visualization of the training process with different augmentations.
• Figure 3: (a)(b) The average attention distance of ViT backbone in multi-heads (dots) w.r.t each layer. (c)(d) The comparison of similarity and variance between discarded 196 Patch Tokens in the output of the ViT backbone.
• Figure 4: An overview of our method, which enhanced feature alignment from both intra- and inter-sample perspectives. We propose the MSTR Module to further improve the Encoder’s feature representation ability by recycling discarded Patch Tokens.
• Figure 5: Comparison of our dataset with other datasets.