Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai

TL;DR

This work addresses the trade-off between semantic alignment and spatial sensitivity in self-supervised vision learning by introducing SiameseIM, a two-branch Siamese framework that predicts dense representations of one augmented view from a masked, differently augmented view of the same image. By applying a single dense UniGrad loss and utilizing relative positional cues between views, SiameseIM achieves strong semantic alignment and fine-grained spatial modeling without relying on multiple, competing loss terms. Empirically, it surpasses both ID and MIM baselines across ImageNet finetuning, linear probing, COCO/LVIS detection, ADE20k segmentation, and robustness tasks, with pronounced gains in few-shot and long-tail settings. The results suggest that dense view-to-view prediction, guided by relative geometry, can unify the advantages of ID and MIM while offering practical performance benefits.

Abstract

Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released at https://github.com/fundamentalvision/Siamese-Image-Modeling.

Siamese Image Modeling for Self-Supervised Vision Representation Learning

TL;DR

This work addresses the trade-off between semantic alignment and spatial sensitivity in self-supervised vision learning by introducing SiameseIM, a two-branch Siamese framework that predicts dense representations of one augmented view from a masked, differently augmented view of the same image. By applying a single dense UniGrad loss and utilizing relative positional cues between views, SiameseIM achieves strong semantic alignment and fine-grained spatial modeling without relying on multiple, competing loss terms. Empirically, it surpasses both ID and MIM baselines across ImageNet finetuning, linear probing, COCO/LVIS detection, ADE20k segmentation, and robustness tasks, with pronounced gains in few-shot and long-tail settings. The results suggest that dense view-to-view prediction, guided by relative geometry, can unify the advantages of ID and MIM while offering practical performance benefits.

Abstract

Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released at https://github.com/fundamentalvision/Siamese-Image-Modeling.
Paper Structure (23 sections, 4 equations, 3 figures, 9 tables)

This paper contains 23 sections, 4 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Augmented Inputs
  5. Prediction Targets
  6. Loss Function
  7. Discussion
  8. Experiments
  9. Implementation Details
  10. Main Results
  11. Ablation Study
  12. Conclusion
  13. Acknowledgments
  14. Implementation Details
  15. Pre-training
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Comparisons among ID, MIM and SiameseIM. Matching different augmented views can help to learn semantic alignment, which is adopted by ID and SiameseIM. Predicting dense representations from masked images is beneficial to obtain spatial sensitivity, which is adopted by MIM and SiameseIM.
  • Figure 2:
  • Figure 3: Positional embedding for online decoder. The positions are calculated with respect to the left-top origin of $x_a$.