PPMStereo: Pick-and-Play Memory Construction for Consistent Dynamic Stereo Matching

TL;DR

PPMStereo tackles temporal inconsistency in dynamic stereo matching by introducing Pick-and-Play Memory (PPM), a compact, query-adaptive memory buffer built through a two-stage process that selects the most informative reference frames and weighs them for spatio-temporal aggregation. A Quality Assessment Module (QAM) computes confidence and redundancy-aware relevance scores to drive top-$K$ frame selection, while an attention-based read-out and RAFT-style iterative refinement fuse memory cues with current features to regress disparities. The approach achieves state-of-the-art temporal consistency and accuracy on challenging benchmarks like Sintel and Dynamic Replica, with favorable computational efficiency compared to prior long-range methods. This work generalizes long-range temporal modeling to dynamic stereo while maintaining practicality for real-world applications, and paves the way for memory-based, scalable dynamic vision systems.

Abstract

Temporally consistent depth estimation from stereo video is critical for real-world applications such as augmented reality, where inconsistent depth estimation disrupts the immersion of users. Despite its importance, this task remains challenging due to the difficulty in modeling long-term temporal consistency in a computationally efficient manner. Previous methods attempt to address this by aggregating spatio-temporal information but face a fundamental trade-off: limited temporal modeling provides only modest gains, whereas capturing long-range dependencies significantly increases computational cost. To address this limitation, we introduce a memory buffer for modeling long-range spatio-temporal consistency while achieving efficient dynamic stereo matching. Inspired by the two-stage decision-making process in humans, we propose a \textbf{P}ick-and-\textbf{P}lay \textbf{M}emory (PPM) construction module for dynamic \textbf{Stereo} matching, dubbed as \textbf{PPMStereo}. PPM consists of a `pick' process that identifies the most relevant frames and a `play' process that weights the selected frames adaptively for spatio-temporal aggregation. This two-stage collaborative process maintains a compact yet highly informative memory buffer while achieving temporally consistent information aggregation. Extensive experiments validate the effectiveness of PPMStereo, demonstrating state-of-the-art performance in both accuracy and temporal consistency. % Notably, PPMStereo achieves 0.62/1.11 TEPE on the Sintel clean/final (17.3\% \& 9.02\% improvements over BiDAStereo) with fewer computational costs. Codes are available at \textcolor{blue}{https://github.com/cocowy1/PPMStereo}.

Figures (14)

• Figure 1: Comparison between prior methods (a) and our method (b). For the $t$-th frame, prior works process video sequences using small temporal sliding windows with attention or optical flow, restricting cost information propagation. Our method captures long-range spatio-temporal relationships across the input sequence by constructing and updating a compact memory buffer.
• Figure 2: An overview of PPMStereo. The gray part is the memory 'pick' process, and the blue part is the memory play process. Our PPMStereo employs a dynamic memory buffer for modeling long-range spatio-temporal relationships while maintaining computational efficiency.
• Figure 3: The details of our Pick-and-play Memory Construction Process (PPM).
• Figure 4: Qualitative comparisons on the Sintel final dataset.
• Figure 5: Temporal consistency comparison on 50-frame reconstructed stereo video (all trained on DR + SF). Our method achieves lower variance, demonstrating superior consistency.