Flow Intelligence: Robust Feature Matching via Temporal Signature Correlation

TL;DR

This work addresses robust feature matching across video streams, including cross-modal settings, where appearance-based methods falter. It introduces Flow Intelligence, a temporal-signature framework that discards spatial keypoints in favor of motion-state sequences derived from per-frame block differences, using a coarse-to-fine, patch-based architecture with quadtree optimization and propagation, all without any training data ($0$ training). The approach comprises Motion State Sequence Construction, Motion Sequence Matching with a binary-sequence distance, and Optimization/Propagation to achieve dense, cross-view correspondences, demonstrated on multi-view and cross-modal datasets with competitive or superior density and speed compared with learning-based baselines; patch-level accuracy remains high even under challenging conditions. Results show strong robustness to viewpoint changes, lighting variation, and modality gaps, suggesting practical impact for real-time, multimodal video understanding without the need for large training corpora.

Abstract

Feature matching across video streams remains a cornerstone challenge in computer vision. Increasingly, robust multimodal matching has garnered interest in robotics, surveillance, remote sensing, and medical imaging. While traditional rely on detecting and matching spatial features, they break down when faced with noisy, misaligned, or cross-modal data. Recent deep learning methods have improved robustness through learned representations, but remain constrained by their dependence on extensive training data and computational demands. We present Flow Intelligence, a paradigm-shifting approach that moves beyond spatial features by focusing on temporal motion patterns exclusively. Instead of detecting traditional keypoints, our method extracts motion signatures from pixel blocks across consecutive frames and extract temporal motion signatures between videos. These motion-based descriptors achieve natural invariance to translation, rotation, and scale variations while remaining robust across different imaging modalities. This novel approach also requires no pretraining data, eliminates the need for spatial feature detection, enables cross-modal matching using only temporal motion, and it outperforms existing methods in challenging scenarios where traditional approaches fail. By leveraging motion rather than appearance, Flow Intelligence enables robust, real-time video feature matching in diverse environments.

Figures (16)

• Figure 1:
• Figure 2: Overview of the Flow Intelligence architecture. Matching is performed from coarse to fine scales by iteratively comparing temporal motion signatures to refine correspondences. State Sequence Construction Module extracts temporal motion patterns from each pixel block. Correlation Computation Module matches blocks by evaluating correlations between their state sequences. Optimization and Propagation Module refines and expands these initial matches.
• Figure 3: Accuracy curves of different feature matching methods, showing performance variations with increasing error threshold.(a) Evaluated on CityData-Aug; (b) Evaluated on OTCBVS-Aug. Note that due to the lack of ground truth labels, the accuracy on OTCBVS-Aug dataset is computed using $MINIMA_{LoFTR}$ as pseudo labels.
• Figure 4: Distribution of matching accuracy, error threshold, and cumulative percentage of confidence-ranked matches for Flow Intelligence.
• Figure 5: Feature matching results on CityData-MV between SIFT, MINIMA-LoFTR and Flow Intelligence (Ours).