Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?

Apratim Bhattacharyya, Bicheng Xu, Sanjay Haresh, Reza Pourreza, Litian Liu, Sunny Panchal, Pulkit Madan, Leonid Sigal, Roland Memisevic

TL;DR

The paper tackles the challenge of enabling live, interactive step-by-step guidance from multi-modal LLMs by introducing the Qualcomm Interactive Cooking benchmark (an extension of CaptainCook4D) with precisely timed instructions and mistake feedback, plus LiveMamba, a streaming vision-language model. It presents a two-stage training regime and novel data augmentations to teach mistake recognition and dynamic re-planning, and provides comprehensive evaluations across zero-shot and fine-tuned regimes, including a turn-based analysis. The results show that current models struggle with real-time, reactive guidance, while LiveMamba substantially improves instruction completion detection and mistake handling, setting a strong baseline for live, situated coaching in cooking tasks.

Abstract

Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.

Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?

TL;DR

The paper tackles the challenge of enabling live, interactive step-by-step guidance from multi-modal LLMs by introducing the Qualcomm Interactive Cooking benchmark (an extension of CaptainCook4D) with precisely timed instructions and mistake feedback, plus LiveMamba, a streaming vision-language model. It presents a two-stage training regime and novel data augmentations to teach mistake recognition and dynamic re-planning, and provides comprehensive evaluations across zero-shot and fine-tuned regimes, including a turn-based analysis. The results show that current models struggle with real-time, reactive guidance, while LiveMamba substantially improves instruction completion detection and mistake handling, setting a strong baseline for live, situated coaching in cooking tasks.

Abstract

Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.

Paper Structure

This paper contains 24 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Qualcomm Interactive Cooking Benchmark and Dataset
  4. LiveMamba for Step-by-Step Instructions
  5. LiveMamba Architecture
  6. LiveMamba Training
  7. LiveMamba Augmentation
  8. Experiments
  9. Experiments
  10. Evaluation Metrics.
  11. Zero-Shot Evaluation
  12. Evaluation of Fine-tuned Models
  13. Turn-based Evaluation
  14. Conclusion
  15. Overview
  16. ...and 9 more sections

Figures (7)

  • Figure 1: An overview of the step-by-step task guidance scenario in our Qualcomm Interactive Cooking benchmark and dataset, where the multi-modal LLM provides instructions and feedback that are sufficient to guide the user towards the goal, e.g., making a tomato mozzarella salad (above).
  • Figure 2: Our LiveMamba model architecture. The input video stream is processed by an InternViT vision head which produces $M$ tokens, and is then reduced to $K$ tokens by a Q-Former. The language backbone produces feedback and invokes the Re-planner if necessary before the next instruction.
  • Figure 3: Our LiveMamba is able to successfully recognize the person has added the black pepper as instructed and points out when the person should heat the oil in a non-stick frying pan, in the Qualcomm Interactive Cooking benchmark.
  • Figure 4: Data samples from the main set. Left: the user prepares spicy tuna avocado wraps. Right: the user prepares spiced hot chocolate.
  • Figure 5: Data samples from the advanced planning set. Left: the user is making ramen. Right: the user is preparing scrambled eggs.
  • ...and 2 more figures