Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling

TL;DR

Procedural steps in egocentric assembly videos are difficult to recognize due to occlusions. The authors propose STORM-PSR, a dual-stream framework that combines assembly state detection with a spatio-temporal transformer, supported by weakly supervised KFS pretraining and KCAS sampling. The method directly predicts correctly completed steps and achieves state-of-the-art or benchmark performance on IndustReal and MECCANO, notably reducing the average recognition delay by up to 26.1%. The work demonstrates that occlusion-resilient temporal modeling can substantially improve PSR performance in real-world industrial tasks, with considerations for computation and data requirements.

Abstract

Procedure step recognition (PSR) aims to identify all correctly completed steps and their sequential order in videos of procedural tasks. The existing state-of-the-art models rely solely on detecting assembly object states in individual video frames. By neglecting temporal features, model robustness and accuracy are limited, especially when objects are partially occluded. To overcome these limitations, we propose Spatio-Temporal Occlusion-Resilient Modeling for Procedure Step Recognition (STORM-PSR), a dual-stream framework for PSR that leverages both spatial and temporal features. The assembly state detection stream operates effectively with unobstructed views of the object, while the spatio-temporal stream captures both spatial and temporal features to recognize step completions even under partial occlusion. This stream includes a spatial encoder, pre-trained using a novel weakly supervised approach to capture meaningful spatial representations, and a transformer-based temporal encoder that learns how these spatial features relate over time. STORM-PSR is evaluated on the MECCANO and IndustReal datasets, reducing the average delay between actual and predicted assembly step completions by 11.2% and 26.1%, respectively, compared to prior methods. We demonstrate that this reduction in delay is driven by the spatio-temporal stream, which does not rely on unobstructed views of the object to infer completed steps. The code for STORM-PSR, along with the newly annotated MECCANO labels, is made publicly available at https://timschoonbeek.github.io/stormpsr .

Figures (8)

• Figure 1: Diagram of the proposed Spatio-Temporal Occlusion-Resilient Modeling for Procedure Step Recognition (STORM-PSR) approach, using assembly state detection for non-occluded frames and temporal modeling to recognize step completions when the assembly object is only partially visible.
• Figure 2: Overview of the spatio-temporal stream of STORM-PSR and its training procedure. The frames of each video clip are passed through the spatial encoder $f_{\text{spatial}}(\cdot)$ to obtain per-frame embeddings. These embeddings are combined with a learnable [CLS] token, and augmented with 1-D positional encoding before being fed into the temporal encoder $f_{\text{temporal}}(\cdot)$. The final state of the [CLS] token provides a clip-level representation, which is fed into the MLP classification head that makes the final predictions. Because multiple procedural steps can be completed within a single clip, the MLP performs multi-label classification. During the training of the temporal encoder, the weights of $f_{\text{spatial}}(\cdot)$ can be either frozen or fine-tuned. The cross-entropy loss is used to optimize the model weights.
• Figure 3: Principle view of generating the probability mass function with bimodal sampling used in key-clip aware sampling (KCAS). The KCAS over-samples hard-negative clips (directly before a step completion occurs) using the first Gaussian of the bimodal distribution, and positive clips (directly after the step completion) with the second Gaussian. Potentially ambiguous frames at the exact moment of step completions are sampled at a reduced rate, as well as frames where no steps are completed. The long tails of the distribution deliberately suppress background clips far from any completion, preventing the mini-batch from being dominated by uninformative content.
• Figure 4: Overview of the key-frame sampling (KFS) weakly supervised pre-training. Each mini-batch consists of real-world images sampled around the time stamps of the PSR labels from different videos, with a sampling window $t_{\text{f}}$. When CAD data are available, synthetic images may be included. The sampled images are passed through a spatial encoder $f_{\text{spatial}}(\cdot)$, comprising a ViT-S VIT architecture, into representation space $H$, after which a three-layer MLP $g(\cdot)$ projects the embeddings into another space $Z$, on which the supervised contrastive loss SupCL is calculated.
• Figure 5: Definition of the 17 toy-motorcycle model assembly components in MECCANO.