Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers

Jonas Römer, Timo Dickscheid

TL;DR

This study investigates whether blockwise self-supervised learning (BWSSL) can train masked video transformers without end-to-end backprop, addressing spatiotemporal dependencies in video data. By partitioning a VideoMAE-style ViT into gradient-isolated blocks and applying local masked reconstruction losses, BWSSL achieves representations that nearly match end-to-end baselines on downstream tasks like linear probing and retrieval across model sizes and block granularities. Depth-resolved analyses reveal that BWSSL shifts high-level decodability to earlier blocks while later blocks exhibit saturation and more geometry-preserving updates, with token mixing and patch-token redistribution emerging in distinctive ways. The work positions BWSSL as a viable alternative to E2E under certain regime conditions, highlighting design choices around decoders, block granularity, and the balance between local supervision and global representation alignment.

Abstract

End-to-end backpropagation couples all layers through a global error signal, enabling coordinated learning but requiring long-range credit assignment. Motivated by recent progress in blockwise self-supervised learning (BWSSL), we ask whether masked video transformers can be trained without end-to-end backpropagation. Applying BWSSL to masked video modeling remains relatively underexplored and must handle spatiotemporal context and long-range temporal structure. More broadly, analyses that compare BWSSL and end-to-end training in terms of learning dynamics and depth-wise representation development remain sparse. We apply blockwise learning to a masked autoencoding video vision transformer by partitioning the encoder into blocks, each of which is optimized with a local masked reconstruction loss. Across model sizes and partition granularities, training converges and yields representations close to matched end-to-end baselines under linear-probe and retrieval proxies. In order to compare intermediate representations, we analyze depth-wise decodability, inter-block similarity, and patch-level diagnostics. Blockwise training exposes higher-level structure earlier, while later blocks saturate and operate in a more geometry-preserving regime. It can also induce token-level shifts consistent with stronger early mixing that pooled metrics can miss. These findings point to late-block saturation and interface formation as contributors to the remaining gap.

Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers

TL;DR

This study investigates whether blockwise self-supervised learning (BWSSL) can train masked video transformers without end-to-end backprop, addressing spatiotemporal dependencies in video data. By partitioning a VideoMAE-style ViT into gradient-isolated blocks and applying local masked reconstruction losses, BWSSL achieves representations that nearly match end-to-end baselines on downstream tasks like linear probing and retrieval across model sizes and block granularities. Depth-resolved analyses reveal that BWSSL shifts high-level decodability to earlier blocks while later blocks exhibit saturation and more geometry-preserving updates, with token mixing and patch-token redistribution emerging in distinctive ways. The work positions BWSSL as a viable alternative to E2E under certain regime conditions, highlighting design choices around decoders, block granularity, and the balance between local supervision and global representation alignment.

Abstract

End-to-end backpropagation couples all layers through a global error signal, enabling coordinated learning but requiring long-range credit assignment. Motivated by recent progress in blockwise self-supervised learning (BWSSL), we ask whether masked video transformers can be trained without end-to-end backpropagation. Applying BWSSL to masked video modeling remains relatively underexplored and must handle spatiotemporal context and long-range temporal structure. More broadly, analyses that compare BWSSL and end-to-end training in terms of learning dynamics and depth-wise representation development remain sparse. We apply blockwise learning to a masked autoencoding video vision transformer by partitioning the encoder into blocks, each of which is optimized with a local masked reconstruction loss. Across model sizes and partition granularities, training converges and yields representations close to matched end-to-end baselines under linear-probe and retrieval proxies. In order to compare intermediate representations, we analyze depth-wise decodability, inter-block similarity, and patch-level diagnostics. Blockwise training exposes higher-level structure earlier, while later blocks saturate and operate in a more geometry-preserving regime. It can also induce token-level shifts consistent with stronger early mixing that pooled metrics can miss. These findings point to late-block saturation and interface formation as contributors to the remaining gap.
Paper Structure (36 sections, 2 equations, 5 figures)

This paper contains 36 sections, 2 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gradient-isolated blockwise learning and BWSSL.
  4. Supervised BWL and reconstruction-style auxiliaries.
  5. Depth-wise representation analysis in blockwise models and ViTs.
  6. Blockwise self-supervised training for video data
  7. Training setup
  8. Training objective
  9. Network architecture
  10. Training data
  11. Comparative Representation Analysis
  12. Downstream task performance metrics
  13. Feature complexity analysis
  14. Feature similarity across network depth
  15. Patch-level detail retention and robustness
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Training and Evaluation of the BWSSL model for video data. We train a ViT for video data using a masked autoencoding objective. The architecture is split into four (this Figure) or six blocks each containing multiple transformer layers. BWSSL training is carried out in sequential & simultaneous fashion, with MSE reconstruction losses attached to block outputs. This is compared to a classical E2E training using the last block output. We analyze learned representations by comparing and probing embeddings generated by each block under BWSSL and E2E training.
  • Figure 2: Downstream task metrics. Results for BWSSL ($K{=}4,6$; sequential vs. simultaneous) and matched E2E baselines, for DeiT-Tiny and DeiT-Small on UCF101. (a) Linear-probe accuracy. (b) Retrieval mAP. (c) Masked reconstruction MSE.
  • Figure 3: Linear probing across increasing target complexity. We probe embeddings from each block using labels that range from (a) low-level stimulus parameters to (b) cross-field relations to (c) high-level action categories. Bottom: example frames from each dataset.
  • Figure 4: Reconstruction loss and inter-block similarity. (a) Masked reconstruction MSE after each block (E2E at the final block). (b) CKA between successive blocks. Lower values indicate larger representational updates. (c) Relationship between inter-block CKA and the corresponding change in retrieval mAP.
  • Figure 5: Patch-level robustness and token mixing across blocks. We quantify (a,b) robustness of linear decoding under extreme spatial occlusion (OccDrop in Sec. \ref{['sec:analysis:patch_level']}) and (c) patch-token homogenization via cosine patch similarity (CPS).