What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction

TL;DR

The paper tackles the challenge of real-time, proactive, situated interaction in vision-language models by using live fitness coaching as a controlled real-world domain. It introduces the Qualcomm Exercise Videos Dataset (Qevd), including Qevd-Fit-300K and Qevd-Fit-Coach, with fine-grained action labels, fitness questions, and corrective feedback to enable grounded, streaming interactions. It proposes Stream-VLM, a streaming vision-language model with a 3D-CNN vision backbone and a LLaMA-2-7B LM, augmented with special tokens to decide when to provide feedback, trained in three stages to support end-to-end interaction. Experiments show that existing VLMs struggle in asynchronous feedback scenarios, while Stream-VLM improves temporal precision and feedback quality, highlighting the potential of end-to-end streaming approaches for grounded, real-time guidance; the work also discusses limitations and ethical considerations for sensitive domains like fitness coaching.

Abstract

Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time.

Figures (7)

• Figure 1: Long-range interactive videos from our Qevd-Fit-Coach benchmark. Live feedbacks provided to the participants are shown below each frame. Corrective feedbacks in red.
• Figure 2: Example annotations available on the short video clips from the Qevd-Fit-300K dataset. Annotations include question/answer pairs from our fitness questions dataset and feedback from a coaching perspective.
• Figure 3: Architecture of the Stream-VLM model. The visual stream is processed by a 3D CNN and the language backbone is a LLaMA-2-7B model; special action tokens (<next> and <feedback>) are highlighted in orange.
• Figure 4: Predicted feedbacks on the Fit-Coach benchmark. The "turn-based" LLaMA-VID and LLaVA-NeXT models are unable to provide corrective feedback and instead generate overly generic and repetitive feedback. The Stream-VLM model has learned to provide relevant feedback at the appropriate time.
• Figure 5: Additional example annotations available on the short video clips from the Qevd-Fit-300K dataset (see also \ref{['fig:fitness_qa', 'fig:finetune_dataset']} in the main paper).