From Videos to Conversations: Egocentric Instructions for Task Assistance

TL;DR

HowToDIV tackles the scarcity of scalable, multimodal conversational data for AR-based procedural guidance by introducing a fully automatic Monologue-to-Dialogue Conversion (MDC) pipeline that turns egocentric instructional videos into two-person expert–novice dialogues with turn-level grounding. Built on NIV and EgoPER, HowToDIV encompasses 507 sessions, 6,636 dialogue turns, and about 24 hours of video across cooking, mechanical repair, and planting, with annotated atomic-step instructions and user errors. Baseline benchmarks using Gemma-3 and Qwen-2.5 are reported with BLEU, ROUGE, and LLM-as-a-Judge scores, highlighting the benefits of explicit step structure for reducing hallucinations and improving grounding in procedural guidance. The dataset and method enable scalable development and evaluation of AR assistants capable of real-time, interactive guidance grounded in egocentric video, advancing practical task-automation research.

Abstract

Many everyday tasks, ranging from appliance repair and cooking to car maintenance, require expert knowledge, particularly for complex, multi-step procedures. Despite growing interest in AI agents for augmented reality (AR) assistance, progress remains limited by the scarcity of large-scale multimodal conversational datasets grounded in real-world task execution, in part due to the cost and logistical complexity of human-assisted data collection. In this paper, we present a framework to automatically transform single person instructional videos into two-person multimodal task-guidance conversations. Our fully automatic pipeline, based on large language models, provides a scalable and cost efficient alternative to traditional data collection approaches. Using this framework, we introduce HowToDIV, a multimodal dataset comprising 507 conversations, 6,636 question answer pairs, and 24 hours of video spanning multiple domains. Each session consists of a multi-turn expert-novice interaction. Finally, we report baseline results using Gemma 3 and Qwen 2.5 on HowToDIV, providing an initial benchmark for multimodal procedural task assistance.

Figures (9)

• Figure 1: Overview of the proposed MDC (Monologue-to-Dialogue Conversion) pipeline. An instructional monologue video with expert subtitles is processed to infer procedural instructions, generate multi-turn expert-user dialogues using LLMs, and temporally align video segments with user turns.
• Figure 2: Distribution of HowToDIV across procedural activities. Left: Proportion of user-expert turns for various tasks. Right: Average conversation length as turn count (top of bars); the bars show conversation count, video duration, and QA pair count.
• Figure 3: Distribution of user dialogue lengths across different speech and activity categories. Category 1—Concise speech with correct actions (avg. 3.4 words); Category 2—Regular (or more verbose) speech with correct actions (avg. 10.9 words); Category 3—Regular speech with user errors (avg. 10 words).
• Figure 4: Histogram of user-turn video lengths, with an average duration of 12.48 seconds.
• Figure 5: The input to our approach can be Top-left: an Instructional video with narration OR Bottom-left: an Instructional video (without narration) and step details. These are processed through the three step approach generating a conversation (with egocentric videos) between user and expert helper.