ProSkill: Segment-Level Skill Assessment in Procedural Videos

TL;DR

ProSkill addresses the lack of large-scale, fine-grained skill benchmarks for procedural videos by introducing a segment-level annotation framework that combines Swiss Tournament-based pair selection, crowd-sourced pairwise judgments, and Elo-based aggregation to produce absolute skill scores. The dataset aggregates 1135 clips across 71 actions (~14 hours) from five real-world sources and provides both absolute and pairwise annotations, enabling regression, ranking, and classification tasks. Absolute scores are updated via $R'_A = R_A + K(S_A - E_A)$ with $E_A = 1/(1 + 10^{(R_B - R_A)/400})$, illustrating the core aggregation mechanism. The work reveals that state-of-the-art skill assessment methods struggle on realistic, multi-action procedural videos, underlining the need for models that integrate robust temporal modeling with fine-grained action-aware scoring.

Abstract

Skill assessment in procedural videos is crucial for the objective evaluation of human performance in settings such as manufacturing and procedural daily tasks. Current research on skill assessment has predominantly focused on sports and lacks large-scale datasets for complex procedural activities. Existing studies typically involve only a limited number of actions, focus on either pairwise assessments (e.g., A is better than B) or on binary labels (e.g., good execution vs needs improvement). In response to these shortcomings, we introduce ProSkill, the first benchmark dataset for action-level skill assessment in procedural tasks. ProSkill provides absolute skill assessment annotations, along with pairwise ones. This is enabled by a novel and scalable annotation protocol that allows for the creation of an absolute skill assessment ranking starting from pairwise assessments. This protocol leverages a Swiss Tournament scheme for efficient pairwise comparisons, which are then aggregated into consistent, continuous global scores using an ELO-based rating system. We use our dataset to benchmark the main state-of-the-art skill assessment algorithms, including both ranking-based and pairwise paradigms. The suboptimal results achieved by the current state-of-the-art highlight the challenges and thus the value of ProSkill in the context of skill assessment for procedural videos. All data and code are available at https://fpv-iplab.github.io/ProSkill/

Figures (9)

• Figure 1: The proposed annotation protocol (top) and key features of the resulting ProSkill dataset (bottom). Stage 1 action video pairs are selected for labeling following a Round of a Swiss Tournament scheme; Stage 2 selected pairs are labeled with a crowdsourcing platform asking users to perform pairwise ranking; Stage 3 the pairwise outcomes are aggregated using an ELO elo1978rating scheme to compute a global leaderboard. Stages 1-3 are iterated for a given number of rounds to achieve stable absolute ratings.
• Figure 2: Screenshot of the annotation interface used in AMT. Annotators view two video segments corresponding to the same task and are asked to select which performer appears more skilled.
• Figure 3: Ranking stability analysis across datasets. (a) Temporal evolution of Kendall’s $\tau$ across consecutive rounds. (b) Comparative round-level stability, highlighting convergence behavior.
• Figure 4: Illustration of the hierarchical structure of the ProSkill dataset, decomposing total annotation time by dataset, overall task type, and individual actions. Segment sizes are proportional to cumulative durations, highlighting dominant activities across different tasks. Action labels are abbreviated for readability, with a side legend offering full descriptions.
• Figure 5: Example of skill ranking for the action Mount the table legs from the IkeaASM dataset. Clips show examples of high (green), medium (yellow), and low (red) ELO-based scores, illustrating differences in efficiency, focus, and interruptions.