Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation

TL;DR

This work tackles the inefficiency of fixed-iteration recurrent optical flow by introducing a context-aware iteration policy that assigns per-sample iteration budgets. It integrates a lightweight policy network with backbones like RAFT, GMA, FlowFormer, and KPA-Flow to decide whether to skip updates, guided by historical hidden state, iteration embedding, and an incremental loss that forecasts future improvements. The policy is controllable via a resource parameter $r$, enabling FLOP reductions of roughly $40\%$ on Sintel and $20\%$ on KITTI while maintaining accuracy. Ablation studies confirm the value of contextual cues and the incremental loss for adapting iterations to sample difficulty, making the approach practical for resource-constrained deployments.

Abstract

Existing recurrent optical flow estimation networks are computationally expensive since they use a fixed large number of iterations to update the flow field for each sample. An efficient network should skip iterations when the flow improvement is limited. In this paper, we develop a Context-Aware Iteration Policy Network for efficient optical flow estimation, which determines the optimal number of iterations per sample. The policy network achieves this by learning contextual information to realize whether flow improvement is bottlenecked or minimal. On the one hand, we use iteration embedding and historical hidden cell, which include previous iterations information, to convey how flow has changed from previous iterations. On the other hand, we use the incremental loss to make the policy network implicitly perceive the magnitude of optical flow improvement in the subsequent iteration. Furthermore, the computational complexity in our dynamic network is controllable, allowing us to satisfy various resource preferences with a single trained model. Our policy network can be easily integrated into state-of-the-art optical flow networks. Extensive experiments show that our method maintains performance while reducing FLOPs by about 40%/20% for the Sintel/KITTI datasets.

Figures (10)

• Figure 1: Efficient inference. Our policy network can skip the iteration to determine the optimal number of iterations depending on contextual information. Users are capable of altering the resource preference value $r$ to control computational complexity.
• Figure 2: Three examples of changes in EPE as iteration increases. When the network encounters a bottleneck or the EPE improvement is small, we can reduce the computational complexity by skipping iterations. The EPE of examples B and C has a bottleneck after the 12th and 10th iterations. Compared with example A and B, the EPE improvement of example C from the 6th to 10th iteration is small.
• Figure 3: Statistical observation. X-axis denotes the minimum number of iteration steps to achieve the near best EPE ($||EPE_x - EPE_{best}||<0.01$). The Y-axis denotes the percentage of samples that achieve the near EPE at the step.
• Figure 4: The architecture of the proposed dynamic optical flow network with the proposed context-aware iteration policy network. $\sum$ represents the aggregation described in Equation \ref{['equ:aggregate']}, and we omit the aggregation for $\hat{f}_t$ in this figure. $\times$ denotes multiplication.
• Figure 5: Qualitative comparison on the KITTI-train dataset. In the EPE map, the blue color is better, and the red is worse.