Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation

Edoardo Bianchi, Antonio Liotta

TL;DR

The paper tackles the problem of estimating human proficiency from video by leveraging synchronized egocentric and exocentric viewpoints. It introduces SkillFormer, a parameter-efficient architecture built on TimeSformer augmented with a CrossViewFusion module and adaptive self-calibration, with LoRA-based fine-tuning to minimize trainable parameters. On the EgoExo4D benchmark, SkillFormer achieves state-of-the-art accuracy in exocentric and multi-view settings (46.3% and 47.5%, respectively) while using about 4.5x fewer parameters and 3.75x fewer training epochs than prior baselines. The approach excels in structured activities such as Basketball, Cooking, and Bouldering, demonstrating the value of unified multi-view fusion for fine-grained skill assessment and efficient deployment in resource-constrained environments.

Abstract

Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. Project page at https://edowhite.github.io/SkillFormer

SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation

TL;DR

The paper tackles the problem of estimating human proficiency from video by leveraging synchronized egocentric and exocentric viewpoints. It introduces SkillFormer, a parameter-efficient architecture built on TimeSformer augmented with a CrossViewFusion module and adaptive self-calibration, with LoRA-based fine-tuning to minimize trainable parameters. On the EgoExo4D benchmark, SkillFormer achieves state-of-the-art accuracy in exocentric and multi-view settings (46.3% and 47.5%, respectively) while using about 4.5x fewer parameters and 3.75x fewer training epochs than prior baselines. The approach excels in structured activities such as Basketball, Cooking, and Bouldering, demonstrating the value of unified multi-view fusion for fine-grained skill assessment and efficient deployment in resource-constrained environments.

Abstract

Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. Project page at https://edowhite.github.io/SkillFormer
Paper Structure (15 sections, 4 equations, 2 figures, 3 tables)

This paper contains 15 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Proposed Methodology
  4. Problem Formulation
  5. Video Feature Extraction
  6. CrossViewFusion for Unified Multi-View Integration
  7. Experimental Setup
  8. Dataset
  9. Implementation Details
  10. Results
  11. Overall Accuracy and Efficiency
  12. Scaling Strategy and Parameter Analysis
  13. Per-Domain Accuracy Analysis
  14. Limitations and Future Work
  15. Conclusions

Figures (2)

  • Figure 1: Overview of the SkillFormer architecture. Multi-view video inputs (one egocentric and up to four exocentric) are processed through a shared TimeSformer backbone fine-tuned with LoRA. Features are fused using the CrossViewFusion module and passed to a classification head.
  • Figure 2: Detailed architecture of the CrossViewFusion module. Input features $(B, V, d)$ undergo: (1) Layer normalization per view, (2) Multi-head cross-attention enabling each view to attend to all others, (3) View aggregation via mean pooling, (4) Feed-forward transformation with GELU activation, (5) Learnable gating mechanism $\mathbf{g} = \sigma(\text{Linear}(\mathbf{h}))$ for selective feature modulation, (6) Final projection, and (7) Adaptive self-calibration using learnable statistics to align with classification space.