Hybrid Cost Volume for Memory-Efficient Optical Flow

TL;DR

This paper proposes a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV, and designs a memory-efficient optical flow network, named HCVFlow, which has very low memory usage and outperforms other memory-efficient methods in terms of accuracy.

Abstract

Current state-of-the-art flow methods are mostly based on dense all-pairs cost volumes. However, as image resolution increases, the computational and spatial complexity of constructing these cost volumes grows at a quartic rate, making these methods impractical for high-resolution images. In this paper, we propose a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV. To construct HCV, we first propose a Top-k strategy to separate the 4D cost volume into two global 3D cost volumes. These volumes significantly reduce memory usage while retaining a substantial amount of matching information. We further introduce a local 4D cost volume with a local search space to supplement the local information for HCV. Based on HCV, we design a memory-efficient optical flow network, named HCVFlow. Compared to the recurrent flow methods based the all-pairs cost volumes, our HCVFlow significantly reduces memory consumption while ensuring high accuracy. We validate the effectiveness and efficiency of our method on the Sintel and KITTI datasets and real-world 4K (2160*3840) resolution images. Extensive experiments show that our HCVFlow has very low memory usage and outperforms other memory-efficient methods in terms of accuracy. The code is publicly available at https://github.com/gangweiX/HCVFlow.

Figures (6)

• Figure 1: Qualitative comparisons on the Sintel test setsintel. Compared to the notable memory-efficient method Flow1Dflow1d, our approach achieves more accurate flow estimation in low-texture regions.
• Figure 2: Comparisons with Flow1D flow1d on high-resolution ($1080\times1920$) images from DAVISdavis17 dataset. We achieve better results than Flow1Dflow1d when consuming similar memory.
• Figure 3: Overview of our HCVFlow. We obtain feature maps at 1/8 and 1/16 resolutions and construct the Hybrid Cost Volume (HCV) using these feature maps. Specifically, we compute initial cost volumes in both horizontal and vertical directions, followed by obtaining 3D cost volumes through a Top-k strategy. Subsequently, we aggregate these volumes using an aggregation module to obtain the final 3D global cost volume. Additionally, we construct a 4D local cost volume. Finally, we input the the hybrid cost volume and initial flow predictions generated by aggregation module into the ConvGRU module for iterative flow prediction.
• Figure 4: Qualitative comparisons with accuracy-oriented methods on the KITTI test setkitti15. Our novel aggregation module aggregates contextual information to reduce mismatches, thus our method outperforms RAFT and GMA in real-world complex texture-less areas.
• Figure 5: Qualitative comparisons with memory-efficiency method Flow1D on the KITTI test setkitti15. Flow1D fails to accurately predict motion near object edges, while our method can precisely estimate local details.