Local Attention Transformers for High-Detail Optical Flow Upsampling

TL;DR

This work targets the bottleneck of convex upsampling in $1/8$-scale optical flow by reframing it as local attention and introducing a Transformer-based convex upsampler (TCU). It decouples the final upsampler, adds multi-scale context, and enables hierarchical 2x upsampling with larger local masks, improving edge alignment and high-detail detail preservation. A data-augmentation strategy (-AUG) is proposed to reduce bilinear interpolation artifacts during training. On FlyingChairs+FlyingThings3D, the method yields notable gains across state-of-the-art models (e.g., Sintel Clean EPE from $1.42$ to $1.26$ for RAFT, $1.31$ to $1.18$ for GMA, and $0.94$ to $0.90$ for FlowFormer), demonstrating that the upsampling design itself can substantially improve high-detail optical-flow performance.

Abstract

Most recent works on optical flow use convex upsampling as the last step to obtain high-resolution flow. In this work, we show and discuss several issues and limitations of this currently widely adopted convex upsampling approach. We propose a series of changes, in an attempt to resolve current issues. First, we propose to decouple the weights for the final convex upsampler, making it easier to find the correct convex combination. For the same reason, we also provide extra contextual features to the convex upsampler. Then, we increase the convex mask size by using an attention-based alternative convex upsampler; Transformers for Convex Upsampling. This upsampler is based on the observation that convex upsampling can be reformulated as attention, and we propose to use local attention masks as a drop-in replacement for convex masks to increase the mask size. We provide empirical evidence that a larger mask size increases the likelihood of the existence of the convex combination. Lastly, we propose an alternative training scheme to remove bilinear interpolation artifacts from the model output. Our proposed ideas could theoretically be applied to almost every current state-of-the-art optical flow architecture. On the FlyingChairs + FlyingThings3D training setting we reduce the Sintel Clean training end-point-error of RAFT from 1.42 to 1.26, GMA from 1.31 to 1.18, and that of FlowFormer from 0.94 to 0.90, by solely adapting the convex upsampler.

Figures (5)

• Figure 1: Current optical flow methods ignore fine details. We show 10 High-detail 64x64 patches from the FlyingThings3D flyingthings dataset and compare a recent baseline (GMA gma) with adding our upsampler. Our upsampling better preserves fine details.
• Figure 2: Original convex optical flow upsampling as proposed by RAFT raft. A high-resolution sub-pixel value is written as a convex combination of the low-resolution input. The convex mask weights are guaranteed to be positive and sum to $1$ by using softmax. This convex upsampling is adopted by the state of the art. We propose improvements based on the observation that the convex combination through a softmax can be rephrased as neighborhood attention hassani2022neighborhood, leading to increased accuracy, which can be used as a drop-in replacement for all existing models.
• Figure 3: Left: the original convex upscaling method as proposed by RAFT raft. Right: our proposed multi-step Transformer convex upsampling network. The feature extractor is adopted from RAFT raft but we extract the features at $3$ all intermediate scales. The Neighborhood Attention Transformer blocks hassani2022neighborhood perform local neighborhood attention to enhance features. Attention is used to both upsample the low-resolution flow as well as the feature maps. The block at the bottom is executed $M$ times and upsamples by factor $2$, and hence for $M=3$ the low-resolution flow is upsampled by factor $8$.
• Figure 4: The average end-point-error for increasing amount of detail on FlyingThings3D (test) flyingthings, after being trained on C+T. The performance degrades fast for higher amount of detail. To give an impression of the level of detail per bucket, $6$ random patches from each bucket are shown for FlyingThings3D flyingthings. The contribution of each level of detail to the overall EPE is given as well. Although it is not provided here, the graph for the Sintel sintel dataset looks almost similar.
• Figure 5: Improvement in end-point-error for increasing amount of detail on FlyingThings3D (test) flyingthings, Sintel Clean (train) and Sintel Final (train) sintel , for a series of our proposed models, after being trained on FlyingChairs (train) flyingchairs_flownet and FlyingThings3D (test) flyingthings.