Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProfVLM: A Lightweight Video-Language Model for Multi-View Proficiency Estimation

Edoardo Bianchi, Jacopo Staiano, Antonio Liotta

TL;DR

ProfVLM introduces a compact vision-language model for multi-view proficiency estimation that jointly predicts a proficiency label and generates expert-like feedback from egocentric and exocentric videos. It uses a frozen TimeSformer encoder, an AttentiveGatedProjector to fuse multi-view features, and a LoRA-tuned SmolLM2 language model to produce text, achieving state-of-the-art accuracy with only 5.3M parameters and 6 training epochs on EgoExo4D. The model reframes proficiency estimation as conditional language generation, enabling transparent, natural-language explanations alongside scores. This work demonstrates the viability and efficiency of integrating visual and linguistic reasoning for skill assessment, with strong semantic alignment of generated feedback and practical coaching applications.

Abstract

Existing approaches to skill proficiency estimation often rely on black-box video classifiers, ignoring multi-view context and lacking explainability. We present ProfVLM, a compact vision-language model that reformulates this task as generative reasoning: it jointly predicts skill level and generates expert-like feedback from egocentric and exocentric videos. Central to our method is an AttentiveGatedProjector that dynamically fuses multi-view features, projected from a frozen TimeSformer backbone into a language model tuned for feedback generation. Trained on EgoExo4D with expert commentaries, ProfVLM surpasses state-of-the-art methods while using up to 20x fewer parameters and reducing training time by up to 60%. Our approach not only achieves superior accuracy across diverse activities, but also outputs natural language critiques aligned with performance, offering transparent reasoning. These results highlight generative vision-language modeling as a powerful new direction for skill assessment.

ProfVLM: A Lightweight Video-Language Model for Multi-View Proficiency Estimation

TL;DR

ProfVLM introduces a compact vision-language model for multi-view proficiency estimation that jointly predicts a proficiency label and generates expert-like feedback from egocentric and exocentric videos. It uses a frozen TimeSformer encoder, an AttentiveGatedProjector to fuse multi-view features, and a LoRA-tuned SmolLM2 language model to produce text, achieving state-of-the-art accuracy with only 5.3M parameters and 6 training epochs on EgoExo4D. The model reframes proficiency estimation as conditional language generation, enabling transparent, natural-language explanations alongside scores. This work demonstrates the viability and efficiency of integrating visual and linguistic reasoning for skill assessment, with strong semantic alignment of generated feedback and practical coaching applications.

Abstract

Existing approaches to skill proficiency estimation often rely on black-box video classifiers, ignoring multi-view context and lacking explainability. We present ProfVLM, a compact vision-language model that reformulates this task as generative reasoning: it jointly predicts skill level and generates expert-like feedback from egocentric and exocentric videos. Central to our method is an AttentiveGatedProjector that dynamically fuses multi-view features, projected from a frozen TimeSformer backbone into a language model tuned for feedback generation. Trained on EgoExo4D with expert commentaries, ProfVLM surpasses state-of-the-art methods while using up to 20x fewer parameters and reducing training time by up to 60%. Our approach not only achieves superior accuracy across diverse activities, but also outputs natural language critiques aligned with performance, offering transparent reasoning. These results highlight generative vision-language modeling as a powerful new direction for skill assessment.

Paper Structure

This paper contains 22 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Proposed Methodology
  4. Problem Formulation
  5. Video Feature Extraction
  6. AttentiveGatedProjector: Multi-View Vision-Language Alignment
  7. Tokenization and LLM Prompt Structure
  8. Experimental Setup
  9. Dataset
  10. Implementation Details
  11. Evaluation Metrics
  12. Results
  13. Baseline Selection Rationale
  14. Overall Classification Accuracy and Efficiency
  15. Cross-Domain Accuracy Analysis
  16. ...and 7 more sections

Figures (2)

  • Figure 1: ProfVLM architecture. Multi-view video is encoded by a frozen TimeSformer, fused via a trainable AttentiveGatedProjector, and decoded by a LoRA-tuned SmolLM2 to generate both proficiency label and commentary.
  • Figure 2: AttentiveGatedProjector architecture. Multi-view input features are initially normalized and combined using multi-head attention. The aggregated representation is then refined through a feed-forward block with residual paths and a trainable gating unit. Finally, the output undergoes linear projection, normalization, and alignment to the language model space via learned scaling and shifting parameters.