EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos

TL;DR

This work introduces EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains and generates multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions.

Abstract

We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains. Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions. We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems. Our evaluation shows that MLLMs achieve close to human-level accuracy on inferring goals from egocentric videos. However, MLLMs (including the largest ones we tested with over 100B parameters) fall short of human performance when inferring the camera wearers' in-the-moment belief states and future actions that are most consistent with the unseen video future. We believe that our results will shape the future design of an important class of egocentric digital assistants which are equipped with a reasonable model of the user's internal mental states.

Figures (5)

• Figure 1: Example questions in EgoToM and difficulty for humans and models, with selected frames before and after the query moment (0s). C indicates the camera wearer. Boldfaced choices are the correct answer best supported by video evidence. For example, in goal inference (right), the single frame at the query moment cannot disambiguate the choices, but the past context suggests that C's main goal is to collect and organize wheat bundles. Some statements were slightly edited and trimmed for visualization purposes.
• Figure 2: Example questions associated with the same query moment in EgoToM. The questions are queried at 3m41s for this instance.
• Figure 3: The EgoToM generation pipeline. Given a query moment, ground-truth goal, belief, and actions are extrapolated using future action narrations. Wrong choices are generated only using narrations prior to the query moment and do not align with the video future.
• Figure 4: Performance of humans, LLM families, and some open-weight VLMs on EgoToM questions. Dashed lines indicate chance level for each question category. Error bars indicate standard error of the mean (for models, this is computed over the mean accuracy for each question over 3 shuffled choice sets). Legends indicate the context provided for the QAs in each model. Humans, open-weight VLMs, and GPT-4-Turbo are evaluated with sample frames from video contexts. Llama models and GPT models are evaluated with text contexts (Ego4D narrations).
• Figure 5: A. Choice consistency between humans and models. B. Error consistency between inference types within each model. Error consistency measures binary answer correctness alignment between two response profiles. Choice consistency measures multi-way choice selection alignment between two response profiles.