Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning

Qi Qian, Yuanhong Xu, Juhua Hu

TL;DR

This work addresses learning with fixed deep features by introducing semantic adversarial augmentation (SeA) in feature space. SeA derives a semantic perturbation direction by projecting the gradient onto a subspace spanned by in batch real features, enabling effective augmentation of fixed representations for a linear classifier. Across 11 downstream tasks and four pre training models, SeA yields about a 2% average improvement over baselines and often matches fine tuning while being far more efficient. The method is further strengthened by an ensemble approach over multiple pre training objectives, and the authors provide thorough ablations and runtime analysis to establish practical applicability.

Abstract

Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. Compared with fine-tuning or linear probing that can explore diverse augmentations, \eg, random crop/flipping, in the original input space, the appropriate augmentations for learning with fixed deep features are more challenging and have been less investigated, which degenerates the performance. To unleash the potential of fixed deep features, we propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Concretely, the adversarial direction implied by the gradient will be projected to a subspace spanned by other examples to preserve the semantic information. Then, deep features will be perturbed with the semantic direction, and augmented features will be applied to learn the classifier. Experiments are conducted on $11$ benchmark downstream classification tasks with $4$ popular pre-trained models. Our method is $2\%$ better than the deep features without SeA on average. Moreover, compared to the expensive fine-tuning that is expected to give good performance, SeA shows a comparable performance on $6$ out of $11$ tasks, demonstrating the effectiveness of our proposal in addition to its efficiency. Code is available at \url{https://github.com/idstcv/SeA}.

SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning

TL;DR

This work addresses learning with fixed deep features by introducing semantic adversarial augmentation (SeA) in feature space. SeA derives a semantic perturbation direction by projecting the gradient onto a subspace spanned by in batch real features, enabling effective augmentation of fixed representations for a linear classifier. Across 11 downstream tasks and four pre training models, SeA yields about a 2% average improvement over baselines and often matches fine tuning while being far more efficient. The method is further strengthened by an ensemble approach over multiple pre training objectives, and the authors provide thorough ablations and runtime analysis to establish practical applicability.

Abstract

Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. Compared with fine-tuning or linear probing that can explore diverse augmentations, \eg, random crop/flipping, in the original input space, the appropriate augmentations for learning with fixed deep features are more challenging and have been less investigated, which degenerates the performance. To unleash the potential of fixed deep features, we propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Concretely, the adversarial direction implied by the gradient will be projected to a subspace spanned by other examples to preserve the semantic information. Then, deep features will be perturbed with the semantic direction, and augmented features will be applied to learn the classifier. Experiments are conducted on benchmark downstream classification tasks with popular pre-trained models. Our method is better than the deep features without SeA on average. Moreover, compared to the expensive fine-tuning that is expected to give good performance, SeA shows a comparable performance on out of tasks, demonstrating the effectiveness of our proposal in addition to its efficiency. Code is available at \url{https://github.com/idstcv/SeA}.
Paper Structure (25 sections, 4 theorems, 18 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 25 sections, 4 theorems, 18 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Features
  4. Augmentation in Feature Space
  5. Semantic Adversarial Augmentation
  6. Semantic Direction
  7. Illustration of Adversarial Direction
  8. Experiments
  9. Ablation Study
  10. Effect of Step Size for Augmentation
  11. Effect of Directions for Augmentation
  12. Comparison with Linear Probing
  13. Running Time
  14. Comparison on Downstream Tasks
  15. Comparison on Ensemble of Features
  16. ...and 10 more sections

Key Result

Proposition 1

With unit length variables $\{\mathbf{x}_j\}$ and $\mathbf{g}_i$, we have

Figures (4)

  • Figure 1: Illustration of semantic adversarial augmentation (SeA). The red solid and empty circles denote the original data and its augmentation, respectively. Left: Conventional adversarial augmentation can perturb with arbitrary direction (e.g., augmentation may appear the same as the original); Right: SeA augments examples with semantic directions spanned by features from real data (e.g., different sectors show different subspaces for augmentation, where we can get more semantic meaningful augmentation).
  • Figure 2: Training accuracy of different $\eta$.
  • Figure 3: Test accuracy of different $\eta$.
  • Figure 4: Test accuracy of various augmentation directions.

Theorems & Definitions (6)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • proof
  • proof