InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Models

TL;DR

InsTALL advances instructional video understanding by fusing online visual streams with a Procedural Graph to guide multi-modal LLMs in real-time. It defines four core tasks—Task Recognition, Action Recognition, Action Prediction, and Plan Prediction—and adds two error-detection objectives, all trained with graphs mined from task videos and integrated into both training and online inference. Across COIN and CrossTask, InsTALL achieves state-of-the-art results, with significant gains when the Procedural Graph is incorporated, demonstrating improved reasoning over sequences and dependencies. The work enables context-aware, streaming assistance for multi-step tasks, with practical impact on visually guided, real-time user support and procedural guidance systems.

Abstract

The improved competence of generative models can help building multi-modal virtual assistants that leverage modalities beyond language. By observing humans performing multi-step tasks, one can build assistants that have situational awareness of actions and tasks being performed, enabling them to cater assistance based on this understanding. In this paper, we develop a Context-aware Instructional Task Assistant with Multi-modal Large Language Models (InsTALL) that leverages an online visual stream (e.g. a user's screen share or video recording) and responds in real-time to user queries related to the task at hand. To enable useful assistance, InsTALL 1) trains a multi-modal model on task videos and paired textual data, and 2) automatically extracts task graph from video data and leverages it at training and inference time. We show InsTALL achieves state-of-the-art performance across proposed sub-tasks considered for multimodal activity understanding -- task recognition (TR), action recognition (AR), next action prediction (AP), and plan prediction (PP) -- and outperforms existing baselines on two novel sub-tasks related to automatic error identification.

Figures (6)

• Figure 1: InsTALL showcasing its ability to understand visual cues of the user's environment and comprehend user instructions to provide context-aware assistance.
• Figure 2: InsTALL comprises of an image encoder, an MLP projector, a temporal aggregator, and an LLM. An input sequence of video frames is processed by the image encoder followed by the MLP. The extracted spatio-temporal features are shown using a grid of image patches across multiple frames, where each frame embedding has $N$ pooled spatial tokens. We then compress $T\times N$ embeddings along the temporal axis. The MLP helps transform these video embeddings to the text space. In addition, InsTALL includes a graph structure constructed from task procedures and language tokens, all input to the LLM.
• Figure 3: Procedural graph$\mathbf{G}$ is a directed graph where nodes are steps of activity and edges are chains of steps that are mined from video data. The graph plays an important role in modeling procedures involved in multi-step tasks to train different instructional understanding objectives (e.g., \ref{['eq:ar_graph']}, \ref{['eq:sr_graph']}, \ref{['eq:sa_graph']}, \ref{['eq:pa_graph']}, \ref{['eq:pap_graph']}, with step and order mistake detection in §\ref{['subsec:mistake']}) and create online streaming dialog.
• Figure 4: Qualitative comparison of our graph-based InsTALL versus VideoLLM-online chen2024videollm for \ref{['eq:sr_graph']}. Our InsTALL is aware of the position of the step in the entire procedure (arrows) and predicts the steps accurately. At the same time, VideoLLM-online misinterprets the order by relying solely on visual cues. Red texts denote incorrect steps while green texts denote correct steps. Best viewed in color.
• Figure : Procedural Graph $\mathbf{G}$ Construction