SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow

TL;DR

SEA-RAFT introduces a simplified, faster RAFT-based optical-flow method that targets high accuracy and efficiency. It advances RAFT with a Mixture-of-Laplace (MoL) loss to handle occlusion and ambiguity, direct regression of an initial flow to accelerate convergence, and large-scale rigid-flow pre-training to boost generalization, all while replacing bespoke RAFT components with standard backbones like ResNet and ConvNeXt. The approach achieves state-of-the-art results on Spring (notably EPE around $3.69$ and 1px around $0.36$) and demonstrates strong cross-dataset generalization to KITTI and Sintel, with substantial speedups (at least $2.3 imes$) and smaller/inference-friendly architectures. These advances yield a practical, high-resolution optical-flow method suitable for real-world applications, with public code to facilitate adoption and further research.

Abstract

We introduce SEA-RAFT, a more simple, efficient, and accurate RAFT for optical flow. Compared with RAFT, SEA-RAFT is trained with a new loss (mixture of Laplace). It directly regresses an initial flow for faster convergence in iterative refinements and introduces rigid-motion pre-training to improve generalization. SEA-RAFT achieves state-of-the-art accuracy on the Spring benchmark with a 3.69 endpoint-error (EPE) and a 0.36 1-pixel outlier rate (1px), representing 22.9% and 17.8% error reduction from best published results. In addition, SEA-RAFT obtains the best cross-dataset generalization on KITTI and Spring. With its high efficiency, SEA-RAFT operates at least 2.3x faster than existing methods while maintaining competitive performance. The code is publicly available at https://github.com/princeton-vl/SEA-RAFT.

Figures (7)

• Figure 1: Zero-shot performance of SEA-RAFT and existing methods on the Spring mehl2023spring training split. Latency is measured on an RTX3090 with a batch size of 1 and input resolution $540\times 960$. SEA-RAFT has an accuracy close to the best one achieved by MS-RAFT+ jahedi2023ms but is $11\times$ smaller and $24\times$ faster.
• Figure 2: Compared with RAFT teed2020raft, SEA-RAFT introduces (1) rigid-flow pre-training, (2) mixture of Laplace loss, and (3) direct regression of initial flow.
• Figure 3: Ambiguous cases can occur frequently in training data where flow is unpredictable due to occlusion. Such cases can dominate the $L_1$ loss (shown as an error map) used by current methods teed2020raftxu2022gmflow. Our new training loss allows the model to account for such uncertainty.
• Figure 4: Visualization on Spring mehl2023spring test set.
• Figure 5: Visualization on Sintel butler2012naturalistic, KITTI menze2015object, and Middlebury baker2011database.