Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Encoder-Decoder Flow Through Shared Structures

Frederik Laboyrie, Mehmet Kerim Yucel, Albert Saa-Garriga

TL;DR

The paper addresses the stagnation of decoder architectures in dense prediction by introducing banks—globally shared tensors that augment each decoding block with cross-block context. Specifically, feature banks $B_f$ and sampling banks $B_s$ are generated by a bank generator $Z$ from the encoder and interact with decoding features via reweighting $X' = X \times \mathrm{conv}(\mathrm{concat}(B, X))$ and guided dynamic upsampling guided by $B_s$ using DySample, with downsampling to decoding resolutions. Empirically, banks yield consistent accuracy gains across ViT-S/B and RepViT on NYUv2 and synthetic datasets, with modest increases in parameters and GFLOPs, and clear qualitative improvements such as reduced artifacts and sharper edges. The approach is presented as a generic add-on to ViT-based dense predictors, with potential applicability to other dense-prediction tasks beyond depth estimation. $B_f$ and $B_s$ provide a unified mechanism to couple encoder features and decoding steps, enabling richer context propagation throughout the decoder pipeline.

Abstract

Dense prediction tasks have enjoyed a growing complexity of encoder architectures, decoders, however, have remained largely the same. They rely on individual blocks decoding intermediate feature maps sequentially. We introduce banks, shared structures that are used by each decoding block to provide additional context in the decoding process. These structures, through applying them via resampling and feature fusion, improve performance on depth estimation for state-of-the-art transformer-based architectures on natural and synthetic images whilst training on large-scale datasets.

Rethinking Encoder-Decoder Flow Through Shared Structures

TL;DR

The paper addresses the stagnation of decoder architectures in dense prediction by introducing banks—globally shared tensors that augment each decoding block with cross-block context. Specifically, feature banks and sampling banks are generated by a bank generator from the encoder and interact with decoding features via reweighting and guided dynamic upsampling guided by using DySample, with downsampling to decoding resolutions. Empirically, banks yield consistent accuracy gains across ViT-S/B and RepViT on NYUv2 and synthetic datasets, with modest increases in parameters and GFLOPs, and clear qualitative improvements such as reduced artifacts and sharper edges. The approach is presented as a generic add-on to ViT-based dense predictors, with potential applicability to other dense-prediction tasks beyond depth estimation. and provide a unified mechanism to couple encoder features and decoding steps, enabling richer context propagation throughout the decoder pipeline.

Abstract

Dense prediction tasks have enjoyed a growing complexity of encoder architectures, decoders, however, have remained largely the same. They rely on individual blocks decoding intermediate feature maps sequentially. We introduce banks, shared structures that are used by each decoding block to provide additional context in the decoding process. These structures, through applying them via resampling and feature fusion, improve performance on depth estimation for state-of-the-art transformer-based architectures on natural and synthetic images whilst training on large-scale datasets.
Paper Structure (12 sections, 7 equations, 4 figures, 4 tables)

This paper contains 12 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Formulation
  5. Banks
  6. Feature Banks
  7. Sampling Banks
  8. Experiments
  9. Training procedure
  10. Results
  11. Ablations
  12. Conclusion

Figures (4)

  • Figure 1: Accuracy ($\delta$1) vs parameters for architectures trained on large-scale dataset and tested on NYUv2. The introduction of our proposed banks can make a ViT-S based model almost match the performance of a ViT-B based model, whilst introducing a negligible amount of parameters.
  • Figure 2: Encoder-decoder flow with banks (a) shown with RefineNet decoder block (b) and our block with banks (c). $\uparrow$ is bilinear upsampling.
  • Figure 3: DySample (a) reconfigured to support guided sampling via reference tensors (b).
  • Figure 4: Qualitative results of RepViT m2_3 on HyperSim dataset. The usage of banks recovers artifacts and improves geometric accuracy of edges.