Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation

Tushar Anand, Maheswar Bora, Antitza Dantcheva, Abhijit Das

TL;DR

DenVisCoM introduces a novel Mamba-based block and a hybrid architecture that jointly estimates optical flow and stereo disparity in real time. By fusing image-pair features inside the Mamba sequence through a DenVisCoM block and a joint Scan pathway, and by integrating self- and cross-attention, the method learns dense visual correspondences efficiently. Empirical results on KITTI, Sintel, and VKITTI show strong accuracy with real-time performance and favorable memory usage, with ablations validating the benefits of joint patch processing and attention. Cross-task pretraining from flow to disparity further improves performance, supporting a unified dense perception model for motion and 3D understanding in robotics and autonomous systems.

Abstract

In this work, we propose a novel Mamba block DenVisCoM, as well as a novel hybrid architecture specifically tailored for accurate and real-time estimation of optical flow and disparity estimation. Given that such multi-view geometry and motion tasks are fundamentally related, we propose a unified architecture to tackle them jointly. Specifically, the proposed hybrid architecture is based on DenVisCoM and a Transformer-based attention block that efficiently addresses real-time inference, memory footprint, and accuracy at the same time for joint estimation of motion and 3D dense perception tasks. We extensively analyze the benchmark trade-off of accuracy and real-time processing on a large number of datasets. Our experimental results and related analysis suggest that our proposed model can accurately estimate optical flow and disparity estimation in real time. All models and associated code are available at https://github.com/vimstereo/DenVisCoM.

DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation

TL;DR

DenVisCoM introduces a novel Mamba-based block and a hybrid architecture that jointly estimates optical flow and stereo disparity in real time. By fusing image-pair features inside the Mamba sequence through a DenVisCoM block and a joint Scan pathway, and by integrating self- and cross-attention, the method learns dense visual correspondences efficiently. Empirical results on KITTI, Sintel, and VKITTI show strong accuracy with real-time performance and favorable memory usage, with ablations validating the benefits of joint patch processing and attention. Cross-task pretraining from flow to disparity further improves performance, supporting a unified dense perception model for motion and 3D understanding in robotics and autonomous systems.

Abstract

In this work, we propose a novel Mamba block DenVisCoM, as well as a novel hybrid architecture specifically tailored for accurate and real-time estimation of optical flow and disparity estimation. Given that such multi-view geometry and motion tasks are fundamentally related, we propose a unified architecture to tackle them jointly. Specifically, the proposed hybrid architecture is based on DenVisCoM and a Transformer-based attention block that efficiently addresses real-time inference, memory footprint, and accuracy at the same time for joint estimation of motion and 3D dense perception tasks. We extensively analyze the benchmark trade-off of accuracy and real-time processing on a large number of datasets. Our experimental results and related analysis suggest that our proposed model can accurately estimate optical flow and disparity estimation in real time. All models and associated code are available at https://github.com/vimstereo/DenVisCoM.
Paper Structure (17 sections, 11 equations, 4 figures, 6 tables)

This paper contains 17 sections, 11 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mamba Architectures
  4. Proposed Methodology
  5. Preliminaries
  6. Proposed DenVisCoM Block
  7. Proposed Architecture
  8. Task Specific Matching
  9. Experiments and Analysis
  10. Optical Flow
  11. Results and Analysis
  12. Ablation on Optical Flow
  13. Disparity task
  14. Results and Analysis
  15. Ablation on disparity task
  16. ...and 2 more sections

Figures (4)

  • Figure 1: The proposed modified DenVisCoM block for sequential joint modeling in the sequence transformation block of Mamba.
  • Figure 2: Overview of the proposed hybrid model.
  • Figure 3: Visual representation of Optical Flow task on Sintel on two samples(left column and right column).
  • Figure 4: Visual representation of Disparity task on Vkitti2 and Sintel.