Removing Cost Volumes from Optical Flow Estimators

TL;DR

This work targets the substantial computational and memory burden of cost volumes in optical flow estimators by introducing ReCoVEr, a training strategy that enables removing cost volumes at inference while preserving accuracy. The approach uses a RAFT-like two-branch architecture with a cost-volume branch and a context/initial-flow branch, coupled with a cut-off training schedule to eliminate the cost-volume path during inference. Three backbones—ReCoVEr-RN, ReCoVEr-CX, and ReCoVEr-MN—cover a range of compute budgets, achieving competitive or state-of-the-art accuracy with significantly reduced runtime and memory (e.g., ReCoVEr-CX near SOTA with 1.2× faster and 6× less memory; ReCoVEr-MN processes Full HD at 20 FPS with ~0.5 GB). The work demonstrates that cost volumes, while beneficial during early training, can be largely dispensed with during inference when the context network provides a reliable initial flow, enabling real-time or resource-constrained optical flow with minimal performance loss.

Abstract

Cost volumes are used in every modern optical flow estimator, but due to their computational and space complexity, they are often a limiting factor regarding both processing speed and the resolution of input frames. Motivated by our empirical observation that cost volumes lose their importance once all other network parts of, e.g., a RAFT-based pipeline have been sufficiently trained, we introduce a training strategy that allows removing the cost volume from optical flow estimators throughout training. This leads to significantly improved inference speed and reduced memory requirements. Using our training strategy, we create three different models covering different compute budgets. Our most accurate model reaches state-of-the-art accuracy while being $1.2\times$ faster and having a $6\times$ lower memory footprint than comparable models; our fastest model is capable of processing Full HD frames at $20\,\mathrm{FPS}$ using only $500\,\mathrm{MB}$ of GPU memory.

Figures (12)

• Figure 1: ReCoVEr. We propose a method to remove cost volumes from optical flow estimators during training, and thereby, we are able to create fast and accurate optical flow estimators with a significantly reduced memory footprint. Our most accurate model, ReCoVEr-CX, reaches state-of-the-art accuracy while being more efficient w.r.t. inference and memory than SEA-RAFT Wang:2024:SEA. Our most efficient model, ReCoVEr-MN, predicts sharper motion boundaries compared to the popular PWC-Net Sun:2018:PWCSun:2020:MMT, while having comparable efficiency.
• Figure 2: Analysis of the computational expense. Cumulative number of floating point operations (FLOPS) required for a single optical flow prediction using SEA-RAFT-M Wang:2024:SEA for various input resolutions. The cost volume of SEA-RAFT is part of the feature network. In this work, we demonstrate a method to remove the feature network during training, thereby eliminating a major part of the required compute operations at inference time.
• Figure 3: ReCoVEr architecture overview. Our method assumes an architecture similar to RAFT Teed:2020:RAP where the input frames are processed by a context network to obtain an initial flow estimate and context features, and in parallel, the inputs are processed by the feature network, consisting of a feature extractor and a cost volume. The outputs of both branches are then combined by the refinement network to obtain the optical flow prediction. Our training strategy allows us to cut away the feature network during training (dotted path). This increases the computational and memory efficiency of the entire optical flow estimator at inference.
• Figure 4: Comparison of the required number of floating point operations (FLOPS) for representative optical flow estimators and our models at various resolutions. Missing data points are due to out-of-memory errors.
• Figure 5: Trade-offs between runtime, memory usage, and accuracies for different models on Spring Mehl:2023:HRH in full resolution.