Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges

TL;DR

This paper surveys vision-based mistake analysis in procedural activities, organizing methods, datasets, and evaluation protocols around how errors are defined, detected, and anticipated. It characterizes mistakes as procedural or executional, and surveys methodological families (structure-free, step-wise, graph-based, template-driven) under varying supervision regimes, including fully supervised, weakly supervised, and unsupervised paradigms. It highlights open challenges such as misalignment between permissible variations and true errors, dataset limitations, and the need for robust error propagation modeling, while pointing to future directions in neuro-symbolic reasoning, video-language supervision, and counterfactual data. The work aims to unify perspectives to advance safe, efficient, and interpretable mistake analysis for real-time human–robot collaboration, automation, and assistive technologies.

Abstract

Mistake analysis in procedural activities is a critical area of research with applications spanning industrial automation, physical rehabilitation, education and human-robot collaboration. This paper reviews vision-based methods for detecting and predicting mistakes in structured tasks, focusing on procedural and executional errors. By leveraging advancements in computer vision, including action recognition, anticipation and activity understanding, vision-based systems can identify deviations in task execution, such as incorrect sequencing, use of improper techniques, or timing errors. We explore the challenges posed by intra-class variability, viewpoint differences and compositional activity structures, which complicate mistake detection. Additionally, we provide a comprehensive overview of existing datasets, evaluation metrics and state-of-the-art methods, categorizing approaches based on their use of procedural structure, supervision levels and learning strategies. Open challenges, such as distinguishing permissible variations from true mistakes and modeling error propagation are discussed alongside future directions, including neuro-symbolic reasoning and counterfactual state modeling. This work aims to establish a unified perspective on vision-based mistake analysis in procedural activities, highlighting its potential to enhance safety, efficiency and task performance across diverse domains.

Figures (6)

• Figure 1: Illustration of mistake analysis in an industrial assembly task: (1) recognize an unexpected or omitted action given an activity protocol (mistake recognition), e.g.,the worker "attached part C" instead of "grabbing part B", (2) temporally detect an error within the execution of an action (mistake detection) and (3) predict a mistake before it fully occurs (early mistake recognition), e.g., warn the worker about selecting the wrong tool. The conceptual worker sketch was generated with google_gemini.
• Figure 2: Illustration of the differences between mistake recognition, early mistake recognition and mistake detection using an example from a filter coffee preparation video. In mistake recognition, the error (e.g., omitting coffee grounds) is identified after the action is completed. In early mistake recognition, the model anticipates the mistake before it fully unfolds (e.g., detect intent to add coffee before coffee filter is added) in the on-going action. In mistake detection the model temporally localizes the specific segment where the mistake occurs. The temporal boundaries of the action and mistake segments can overlap.
• Figure 3: Illustration of the association between video anomaly and mistake detection in procedural activities.Human sketches in the figure generated with openai_chatgpt.
• Figure 4: Hierarchical taxonomy of error types in procedural activities. The taxonomy distinguishes between procedural and executional errors, which arise from incorrect technique, timing, or action semantics. Scenario-specific errors are included to account for domain-dependent cases.
• Figure 5: Timeline of datasets supporting mistake analysis (2018–2025). Circle sizes refer to dataset scale. Datasets: Epic-Tent EPICTENT, CSV CSV, Assembly-101 Assembly101, BRIO-TA BRIOTA, ATA ATA, HoloAssist HoloAssist, CaptainCook4D CaptainCook4D, Ego-Exo4D grauman2022ego4d, IndustReal IndustReal, EgoPER EgoPED and EgoOops EgoOops. Ego4D-M and EK-M li2025mistake are synthetic extensions of Ego4D grauman2022ego4d and EK-100 damen2022rescaling, generated to incorporate explicit mistake annotations.