TRec: Egocentric Action Recognition using 2D Point Tracks
Dennis Holzmann, Sven Wachsmuth
TL;DR
This work tackles egocentric action recognition by introducing 2D point tracks as a lightweight, motion-centric cue. The method, TRec, fuses random point trajectories tracked by CoTracker with RGB frames in a Transformer-based architecture built on a ResNet18 backbone, enabling recognition without hand/object detectors. Experiments show substantial gains over RGB baselines, with background motion also contributing meaningful contextual information; notably, even using only the initial frame with tracks yields competitive accuracy. Overall, the approach demonstrates that explicit motion trajectories can robustly complement appearance cues, offering a scalable alternative for fine-grained egocentric action understanding.
Abstract
We present a novel approach for egocentric action recognition that leverages 2D point tracks as an additional motion cue. While most existing methods rely on RGB appearance, human pose estimation, or their combination, our work demonstrates that tracking randomly sampled image points across video frames can substantially improve recognition accuracy. Unlike prior approaches, we do not detect hands, objects, or interaction regions. Instead, we employ CoTracker to follow a set of randomly initialized points through each video and use the resulting trajectories, together with the corresponding image frames, as input to a Transformer-based recognition model. Surprisingly, our method achieves notable gains even when only the initial frame and its associated point tracks are provided, without incorporating the full video sequence. Experimental results confirm that integrating 2D point tracks consistently enhances performance compared to the same model trained without motion information, highlighting their potential as a lightweight yet effective representation for egocentric action understanding.