ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding

TL;DR

ProMQA introduces a novel evaluation benchmark for multimodal procedural understanding by coupling cooking videos with textual recipes and QA tasks. It employs a Generate-then-Verify annotation pipeline where LLMs generate candidate QA pairs and humans verify and enrich with human-written answers, yielding 401 verified examples from CaptainCook4D. Benchmark results show humans outperform unimodal, Socratic, and multimodal baselines, including competitive proprietary models, highlighting the remaining gap in multimodal procedural understanding. The dataset serves as an evaluation resource for driving progress in practical multimodal systems rather than training data, with careful attention to bias and quality control through adjudication. Limitations include dataset size, domain focus on Western cooking, and the use of evaluation data under permissive licenses.

Abstract

Multimodal systems have great potential to assist humans in procedural activities, where people follow instructions to achieve their goals. Despite diverse application scenarios, systems are typically evaluated on traditional classification tasks, e.g., action recognition or temporal action segmentation. In this paper, we present a novel evaluation dataset, ProMQA, to measure system advancements in application-oriented scenarios. ProMQA consists of 401 multimodal procedural QA pairs on user recording of procedural activities, i.e., cooking, coupled with their corresponding instructions/recipes. For QA annotation, we take a cost-effective human-LLM collaborative approach, where the existing annotation is augmented with LLM-generated QA pairs that are later verified by humans. We then provide the benchmark results to set the baseline performance on ProMQA. Our experiment reveals a significant gap between human performance and that of current systems, including competitive proprietary multimodal models. We hope our dataset sheds light on new aspects of models' multimodal understanding capabilities.

Figures (14)

• Figure 1: Illustration of a system supporting a user in a procedural activity. The left graph is the recipe and the columns of images are screenshots of the user's actions in chronological order. During the activity, the user makes two mistakes. One is a timing error, where the user sets a longer time than required for microwaving (red). The other is a missing step, where the user skips adding sugar (yellow borders for steps after the missing step). Steps with green borders do not have any errors. QAs are occurring at each divider's position.
• Figure 2: Question approval counts (left) and the answer counts by source (right) for each question type.
• Figure 3: Answer source: The number of examples with only machine-generated answers, only human-written answers, or both types of answers (count).
• Figure 4: Answer type: The number of examples with only direct answers, direct answers and suggestions, direct answers and interventions, only suggestions, or all types of answers (count). Note that other combinations, i.e., only interventions or suggestions and interventions, are not found in our dataset.
• Figure 5: Example prompt with recording steps to embed recording information, an on-target excerpt from a recipe, and a question type for QA generation.