Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models

TL;DR

This work investigates cross-view generalization for fine-grained human activity recognition by systematically evaluating four foundation models (VideoMAE, CLIP, X-CLIP, DinoV2) and 13 temporal fusion strategies. By freezing backbones and applying linear probes on two domain-specific datasets (Drive&Act and IKEA-ASM), the study reveals that image-based foundation models, when equipped with powerful temporal fusion (notably self-attention over all tokens), can surpass video-based foundations in both trained and novel views, with DinoV2 frequently achieving the best results. The findings highlight that temporal fusion choice is a critical factor for fine-grained HAR and that cross-view shifts degrade performance, motivating further work on view-invariant representations. The paper provides practical guidance for backbone and fusion method selection and suggests that advanced temporal modeling substantially enhances cross-view fine-grained HAR using foundation models, with code and models to be released to the community.

Abstract

Foundation models (FMs) are large neural networks trained on broad datasets, excelling in downstream tasks with minimal fine-tuning. Human activity recognition in video has advanced with FMs, driven by competition among different architectures. However, high accuracies on standard benchmarks can draw an artificially rosy picture, as they often overlook real-world factors like changing camera perspectives. Popular benchmarks, mostly from YouTube or movies, offer diverse views but only coarse actions, which are insufficient for use-cases needing fine-grained, domain-specific actions. Domain-specific datasets (e.g., for industrial assembly) typically use data from limited static perspectives. This paper empirically evaluates how perspective changes affect different FMs in fine-grained human activity recognition. We compare multiple backbone architectures and design choices, including image- and video- based models, and various strategies for temporal information fusion, including commonly used score averaging and more novel attention-based temporal aggregation mechanisms. This is the first systematic study of different foundation models and specific design choices for human activity recognition from unknown views, conducted with the goal to provide guidance for backbone- and temporal- fusion scheme selection. Code and models will be made publicly available to the community.

Figures (5)

• Figure 1: A high-level overview of a cross-view human activity recognition framework featuring pretrained frozen Foundation Models (FM) with linear probing and a temporal fusion mechanism. We study fine-grained activity understanding and cross-view generalization of different image- and video-based FMs and implement different techniques for linking temporal frame-level information.
• Figure 2: An overview of the used framework, where the transformer block is depicted on the left hand side, the feature extraction pipeline is in the middle, and the temporal fusion strategies are illustrated on the right.
• Figure 3: Visualization of embeddings on the Drive&Act and IKEA-ASM datasets using different models. (a) DinoV2 with Max Pooling, (b) DinoV2 with Self-Attention only on CLS Token, and (c) DinoV2 with Self-Attention on All Tokens. Each color represents a different view of the dataset.
• Figure 4: Performance on the individual views. The plots show the mean class accuracy of various foundation models in both the trained and novel view settings
• Figure 5: Qualitative results of DinoV2 with Self-attention fusion on different views as well as the Ground Truth (GT) activity. Green check marks and red crosses indicate correct and incorrect predictions, respectively.