User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance

TL;DR

This work assesses how modern multimodal LLMs can serve as vision-powered assistants for multi-step tasks by grounding untrimmed egocentric video histories, forecasting subsequent actions, and replanning with user input. It compares two dominant approaches—Socratic Models (text-grounded history via narrations) and Vision-conditioned Language Models (embedding-based grounding)—across offline benchmarks (LTA on Ego4D and VPA on CrossTask) and an online user study with 18 participants performing cooking tasks using Aria. The findings show Socratic models consistently outperform VCLMs in online settings and for long visual histories, while VCLMs can offer advantages for short histories, especially with smaller LLMs. Crucially, offline metrics do not reliably predict online performance, with grounding errors emerging as the primary barrier to successful real-world assistance. The work outlines necessary directions for grounding, longer visual histories, and online evaluation to advance practical vision-based assistive systems.

Abstract

Our research investigates the capability of modern multimodal reasoning models, powered by Large Language Models (LLMs), to facilitate vision-powered assistants for multi-step daily activities. Such assistants must be able to 1) encode relevant visual history from the assistant's sensors, e.g., camera, 2) forecast future actions for accomplishing the activity, and 3) replan based on the user in the loop. To evaluate the first two capabilities, grounding visual history and forecasting in short and long horizons, we conduct benchmarking of two prominent classes of multimodal LLM approaches -- Socratic Models and Vision Conditioned Language Models (VCLMs) on video-based action anticipation tasks using offline datasets. These offline benchmarks, however, do not allow us to close the loop with the user, which is essential to evaluate the replanning capabilities and measure successful activity completion in assistive scenarios. To that end, we conduct a first-of-its-kind user study, with 18 participants performing 3 different multi-step cooking activities while wearing an egocentric observation device called Aria and following assistance from multimodal LLMs. We find that the Socratic approach outperforms VCLMs in both offline and online settings. We further highlight how grounding long visual history, common in activity assistance, remains challenging in current models, especially for VCLMs, and demonstrate that offline metrics do not indicate online performance.

Figures (8)

• Figure 1: Online deployment and evaluation of multimodal LLMs for vision-powered assistive systems. 1) Users equipped with Aria project_aria, which captures an egocentric video stream of their actions, perform multi-step activities while following assistance from multimodal LLMs. 2) These models encode the video stream using text-based representations and optionally vision embeddings to predict future actions to complete the activity. 3) These actions are replanned during execution to provide real-time assistance.
• Figure 2: Architecture for Socratic and VCLM with few-shot examples. We perform few-shot inference of future actions in our benchmark tasks using untrimmed video history as input. Both Socratic and VCLM convert the video history into text narrations using video narration models (as highlighted in the shaded box). Apart from the narrations, VCLM also embeds the video history as continuous embeddings using a video encoder and Perceiver sampler moon2023anymal. The output is a sequence of natural language sentences, which are then mapped to a closed set of actions for each benchmark.
• Figure 3: Overview of the flow of steps for the activity of making caprese salad. Unassisted actions performed by the participants in the partial progress phase of the activity followed by multimodal LLM assisted activity execution is shown.
• Figure 4: Various error modes of multimodal LLMs in the latte activity. Models fail to ground the steps that are already completed, recommend steps with incorrect ordering (planning error), or fail to recognize that the activity is complete.
• Figure 5: Prompt template for Ego4D LTA. We set $N$ to be the total number of actions in the video and $T$ to be the starting action index that we want to predict in the current video.