Vision Transformers with Patch Diversification

TL;DR

This document functions as the ICCV 2021 submission style guide, detailing language requirements, dual-submission policies, page-length limits, and blind-review procedures. It prescribes a strict two-column LaTeX-based formatting framework, including margins, fonts, figure handling, references, and color usage, and it mandates a signed IEEE copyright release for the final version. The guidelines aim to standardize manuscript formatting, facilitate fair and efficient peer review, and ensure consistent presentation across submissions. By delineating differences between review and camera-ready copies (e.g., page numbering), the guide supports a streamlined publication workflow.

Abstract

Vision transformer has demonstrated promising performance on challenging computer vision tasks. However, directly training the vision transformers may yield unstable and sub-optimal results. Recent works propose to improve the performance of the vision transformers by modifying the transformer structures, e.g., incorporating convolution layers. In contrast, we investigate an orthogonal approach to stabilize the vision transformer training without modifying the networks. We observe the instability of the training can be attributed to the significant similarity across the extracted patch representations. More specifically, for deep vision transformers, the self-attention blocks tend to map different patches into similar latent representations, yielding information loss and performance degradation. To alleviate this problem, in this work, we introduce novel loss functions in vision transformer training to explicitly encourage diversity across patch representations for more discriminative feature extraction. We empirically show that our proposed techniques stabilize the training and allow us to train wider and deeper vision transformers. We further show the diversified features significantly benefit the downstream tasks in transfer learning. For semantic segmentation, we enhance the state-of-the-art (SOTA) results on Cityscapes and ADE20k. Our code is available at https://github.com/ChengyueGongR/PatchVisionTransformer.

Figures (2)

• Figure 1: Example of caption. It is set in Roman so that mathematics (always set in Roman: $B \sin A = A \sin B$) may be included without an ugly clash.
• Figure 2: Example of a short caption, which should be centered.