Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowFormer: A Transformer Architecture for Optical Flow

Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, Hongsheng Li

TL;DR

FlowFormer introduces a transformer-based pipeline for optical flow that operates on a 4D cost volume. It encodes this volume into a compact latent cost memory via an Alternate-Group Transformer and decodes dense flow fields using a recurrent cross-attention decoder with dynamic cost queries. The approach achieves state-of-the-art results on Sintel and strong generalization to KITTI, while ablations validate the effectiveness of latent cost tokens and AGT. Additionally, FlowFormer demonstrates that ImageNet-pretrained transformers can benefit optical-flow estimation. Overall, the method combines cost-volume insights with powerful long-range modeling to improve accuracy and robustness in flow estimation.

Abstract

We introduce optical Flow transFormer, dubbed as FlowFormer, a transformer-based neural network architecture for learning optical flow. FlowFormer tokenizes the 4D cost volume built from an image pair, encodes the cost tokens into a cost memory with alternate-group transformer (AGT) layers in a novel latent space, and decodes the cost memory via a recurrent transformer decoder with dynamic positional cost queries. On the Sintel benchmark, FlowFormer achieves 1.159 and 2.088 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the best published result (1.388 and 2.47). Besides, FlowFormer also achieves strong generalization performance. Without being trained on Sintel, FlowFormer achieves 1.01 AEPE on the clean pass of Sintel training set, outperforming the best published result (1.29) by 21.7%.

FlowFormer: A Transformer Architecture for Optical Flow

TL;DR

FlowFormer introduces a transformer-based pipeline for optical flow that operates on a 4D cost volume. It encodes this volume into a compact latent cost memory via an Alternate-Group Transformer and decodes dense flow fields using a recurrent cross-attention decoder with dynamic cost queries. The approach achieves state-of-the-art results on Sintel and strong generalization to KITTI, while ablations validate the effectiveness of latent cost tokens and AGT. Additionally, FlowFormer demonstrates that ImageNet-pretrained transformers can benefit optical-flow estimation. Overall, the method combines cost-volume insights with powerful long-range modeling to improve accuracy and robustness in flow estimation.

Abstract

We introduce optical Flow transFormer, dubbed as FlowFormer, a transformer-based neural network architecture for learning optical flow. FlowFormer tokenizes the 4D cost volume built from an image pair, encodes the cost tokens into a cost memory with alternate-group transformer (AGT) layers in a novel latent space, and decodes the cost memory via a recurrent transformer decoder with dynamic positional cost queries. On the Sintel benchmark, FlowFormer achieves 1.159 and 2.088 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the best published result (1.388 and 2.47). Besides, FlowFormer also achieves strong generalization performance. Without being trained on Sintel, FlowFormer achieves 1.01 AEPE on the clean pass of Sintel training set, outperforming the best published result (1.29) by 21.7%.
Paper Structure (11 sections, 5 equations, 3 figures, 3 tables)

This paper contains 11 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Building the 4D Cost Volume
  5. Cost Volume Encoder
  6. Cost Memory Decoder for Flow Estimation
  7. Experiment
  8. Quantitative Experiment
  9. Qualitative Experiment
  10. Ablation Study
  11. Conclusion

Figures (3)

  • Figure 1: Architecture of FlowFormer. FlowFormer estimates optical flow in three steps: 1) building a 4D cost volume from image features. 2) A cost volume encoder that encodes the cost volume into the cost memory. 3) A recurrent transformer decoder that decodes the cost memory with the source image context features into flows.
  • Figure 2: Alternate-Group Transformer Layer. The alternate-group transformer layer (AGT) alternatively groups tokens in $\mathbf T$ into $H\times W$ groups that contains $K$ tokens ($\mathbf T_\mathbf x$) and $K$ groups that contains $H\times W$ tokens ($\mathbf T_i$), and encode tokens inside groups via self-attention and ss self-attention chu2021twins respectively.
  • Figure 3: Qualitative comparison on the Sintel test set. FlowFormer greatly reduces the flow leakage around object boundaries (pointed by red arrows) and clearer details (pointed by blue arrows).