TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion

TL;DR

This paper tackles video frame interpolation under fast, nonlinear motion by leveraging high-temporal-resolution event cameras. It introduces TimeTracker, which casts motion estimation as continuous trajectory tracking of scene regions using events, aided by appearance-based region segmentation and a transformer-based tracking module. A frame refinement pipeline and global optical-flow optimization enable accurate intermediate frames, and the authors also contribute CHMD, a real-world dataset with complex motion. Results show state-of-the-art performance on synthetic and real datasets, demonstrating robust interpolation in challenging nonlinear motion scenarios.

Abstract

Video frame interpolation (VFI) that leverages the bio-inspired event cameras as guidance has recently shown better performance and memory efficiency than the frame-based methods, thanks to the event cameras' advantages, such as high temporal resolution. A hurdle for event-based VFI is how to effectively deal with non-linear motion, caused by the dynamic changes in motion direction and speed within the scene. Existing methods either use events to estimate sparse optical flow or fuse events with image features to estimate dense optical flow. Unfortunately, motion errors often degrade the VFI quality as the continuous motion cues from events do not align with the dense spatial information of images in the temporal dimension. In this paper, we find that object motion is continuous in space, tracking local regions over continuous time enables more accurate identification of spatiotemporal feature correlations. In light of this, we propose a novel continuous point tracking-based VFI framework, named TimeTracker. Specifically, we first design a Scene-Aware Region Segmentation (SARS) module to divide the scene into similar patches. Then, a Continuous Trajectory guided Motion Estimation (CTME) module is proposed to track the continuous motion trajectory of each patch through events. Finally, intermediate frames at any given time are generated through global motion optimization and frame refinement. Moreover, we collect a real-world dataset that features fast non-linear motion. Extensive experiments show that our method outperforms prior arts in both motion estimation and frame interpolation quality.

Figures (8)

• Figure 1: Visual comparison of our method with other SOTA methods. (b) and (c) estimate optical flow from images, (d) estimates optical flow from events, and (e) fuses image and event information to estimate optical flow. Our method, based on continuous point tracking for optical flow estimation, achieves the best performance.
• Figure 2: Illustration of (a) four flow-based VFI paradigms and (b) their comparison results. Image-based methods like SuperSlomo superslomo rely on a linear motion assumption, which results in significant inaccuracies in nonlinear motion scenarios. Timelens timelens is a typical event-based VFI method that can only estimate sparse optical flow. TimelensXL timelens-xl iteratively computes any-time optical flow by synthesizing intermediate frames, but errors in synthesized frames directly impact the accuracy of optical flow. We achieve dense any-time optical flow through image region segmentation and event-based point tracking, improving VFI performance in nonlinear scenarios.
• Figure 3: The overall architecture of the TimeTracker includes a Scene-Aware Region Segmentation (SARS) module, a Continuous Trajectory guided Motion Estimation (CTME) module, and a Frame Refinement (FR) module. The SARS segments the scene into multiple similar regions as tracking templates based on motion and appearance information. The CTME tracks the motion trajectories of each region and forms a dense any-time optical flow. Finally, the FR refines the warped images to obtain the interpolation results.
• Figure 4: The correlation between appearance and motion. We select three regions with similar appearances, p0$\sim$p2 , in the image (a) and optical flow (b), along with three regions containing both foreground and background, p3$\sim$p5. (c) and (d) illustrate that small regions with minimal pixel value differences and spatial proximity in the image also exhibit consistent optical flow, and vice versa.
• Figure 5: Two continuous motion estimation paradigms. (a) Traditional motion estimation methods calculate the cost volume between temporally adjacent features. Due to the low spatial distinguishability and sparsity of events, mismatches are likely to occur. (b) Since events are temporally continuous, performing continuous tracking within a local region is more robust.