Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling

TL;DR

A novel system, CARS, is introduced to overcome issues through adaptive video context modeling to better preserve the video context for multi-scale activity recognition through adaptive video context modeling.

Abstract

Video activity recognition has become increasingly important in robots and embodied AI. Recognizing continuous video activities poses considerable challenges due to the fast expansion of streaming video, which contains multi-scale and untrimmed activities. We introduce a novel system, CARS, to overcome these issues through adaptive video context modeling. Adaptive video context modeling refers to selectively maintaining activity-related features in temporal and spatial dimensions. CARS has two key designs. The first is an activity spatial feature extraction by eliminating irrelevant visual features while maintaining recognition accuracy. The second is an activity-aware state update introducing dynamic adaptability to better preserve the video context for multi-scale activity recognition. Our CARS runs at speeds $>$30 FPS on typical edge devices and outperforms all baselines by 1.2\% to 79.7\% in accuracy. Moreover, we explore applying CARS to a large video model as a video encoder. Experimental results show that our CARS can result in a 0.46-point enhancement (on a 5-point scale) on the in-distribution video activity dataset, and an improvement ranging from 1.19\% to 4\% on zero-shot video activity datasets.

Figures (10)

• Figure 1: An example of continuous video activity recognition in streaming video. CARS takes every frame as input and continuously recognizes human activities.
• Figure 2: Difference between our idea and existing solutions on maintaining the hidden state.
• Figure 3: Typical designs for streaming video activity recognition. The "All Frames at Once" design processes all video frames at every moment. The "Sliding Window" design incorporates only the latest (or sampled) $m$ frames. The "State Assistance" relies on a single frame at a time and maintains video context through a hidden state.
• Figure 4: Processing speed of video activity recognition on RTX4090 (a 100k-frame video). Baselines 1(A) and 2(A) represent Transformer and Convpooling-based solutions using the "All Frames at Once" design. Baselines 1(F) and 2(F) employ the "Fixed window size" design (the window size is 100). Baseline 3 denotes an LSTM-based "State Assistance" design.
• Figure 5: Video frames of the balance beam sports, selected from the FineGym99 dataset shao2020finegym. The entire gymnastic movement was captured in only 17 frames, lasting less than 700 milliseconds.