Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges

TL;DR

This work formulates an egocentric procedural AI assistant (EgoProceAssist) designed to support step-by-step tasks from a first-person view, decomposing the problem into three core tasks: error detection, procedural learning, and procedural question answering. It introduces a taxonomy and surveys a wide range of methods, datasets, and evaluation metrics across these domains, including both only-video and multimodal approaches. Through supplementary experiments on multiple datasets, the paper exposes substantial gaps in current vision-language models and AI assistants for real-time, egocentric procedural support. The study highlights critical challenges—data scarcity, long-term temporal reasoning, and real-time capabilities—and outlines concrete directions for developing robust, egocentric procedural intelligence with future datasets, memory-augmented reasoning, and improved multimodal fusion.

Abstract

Driven by recent advances in vision language models (VLMs) and egocentric perception research, we introduce the concept of an egocentric procedural AI assistant (EgoProceAssist) tailored to step-by-step support daily procedural tasks in a first-person view. In this work, we start by identifying three core tasks: egocentric procedural error detection, egocentric procedural learning, and egocentric procedural question answering. These tasks define the essential functions of EgoProceAssist within a new taxonomy. Specifically, our work encompasses a comprehensive review of current techniques, relevant datasets, and evaluation metrics across these three core areas. To clarify the gap between the proposed EgoProceAssist and existing VLM-based AI assistants, we introduce novel experiments and provide a comprehensive evaluation of representative VLM-based methods. Based on these findings and our technical analysis, we discuss the challenges ahead and suggest future research directions. Furthermore, an exhaustive list of this study is publicly available in an active repository that continuously collects the latest work: https://github.com/z1oong/Building-Egocentric-Procedural-AI-Assistant

Figures (8)

• Figure 1: Taxonomy of EgoProceAssist flaborea2024pregoplini2024tihuang2025modelingpatsch2025technicallee2024errording2023everystorks2024transparenthaneji2024egooopsmazzamuto2025gazingbansal2024unitedalayrac2016unsupervisedelhamifar2019unsupervisedkukleva2019unsupervisedchowdhury2024opelsener2015unsupervisedlin2022learningzhou2018towardszhukov2019crossnaing2020procedurebansal2022mymahmood2025procedureshah2023stepswang2023lifelongmemoryzhang2024hcqazhang2025hcqataluzzi2025pixelsyang2025egolifedi2024groundedhuang2024vincisuglia2024alanavlmchen2025groundedbiswas2025ravenye2024mm
• Figure 2: Overall Structure. In Section II, domains and techniques related to the construction of EgoProceAssist are presented. In Sections III, IV, and V, we provide a comprehensive summary of the existing technical approaches, commonly used datasets, and evaluation metrics for the three core tasks, respectively. For enhanced clarity, we also present comparative tables to highlight performance differences among the methods. Section VI presents experimental investigations assessing the capability of existing models to understand procedural tasks across two distinct domains. Section VII discusses the current challenges faced in the field and explores potential trends for future research and development. Finally, Section VIII offers a comprehensive summary of the findings and conclusions drawn from this work.
• Figure 3: A timeline with the surveys in egocentric vision bambach2015surveynguyen2016recognitiondel2016summarizationhamid2017surveyrodin2021predictingchen2021surveynunez2022egocentricbandini2020analysisplizzari2024outlookthatipelli2025egocentricli2025challenges.
• Figure 4: (a) shows the overall framework flowchart summarizing only-video egocentric procedural error detection methods: VQF patsch2025technical, EgoPED lee2024error, TI-PREGO plini2024ti, AMNAR huang2025modeling. (b) shows a flowchart summarizing multimodal-based methods: KRR ding2023every, TAC storks2024transparent, EgoOops haneji2024egooops, GC mazzamuto2025gazing, where some methods use gaze or procedure-related text as additional input.
• Figure 5: (a) shows a flowchart summarizing unsupervised learning methods: GPL bansal2024united, ULNI alayrac2016unsupervised, JointSeqFL elhamifar2019unsupervised, ULAC kukleva2019unsupervised, OPEL chowdhury2024opel, BP-HMM sener2015unsupervised, (b) shows a flowchart summarizing weakly-supervised learning methods: PLDS lin2022learning, ProNets zhou2018towards, CrossTask zhukov2019cross, SPS naing2020procedure.