VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities

TL;DR

This work introduces VHAKG, a novel MMKG built from synchronized multi-view videos of daily activities to capture both event sequences and frame-by-frame changes. It extends VirtualHome-AIST with a rich ontology, embeds video data as literals, and links frame-level bounding boxes to dynamic entities, all represented in RDF for web distribution. The authors further compress the KG by removing redundancy and using MPEG-based video compression, enabling practical sharing and tooling for querying. As a demonstration, they design a VQA-style LVLM benchmark and show that while zero-shot results are weak, few-shot learning yields improvements, validating VHAKG as a useful resource for sim2real tasks and LVLM evaluation.

Abstract

Multi-modal knowledge graphs (MMKGs), which ground various non-symbolic data (e.g., images and videos) into symbols, have attracted attention as resources enabling knowledge processing and machine learning across modalities. However, the construction of MMKGs for videos consisting of multiple events, such as daily activities, is still in the early stages. In this paper, we construct an MMKG based on synchronized multi-view simulated videos of daily activities. Besides representing the content of daily life videos as event-centric knowledge, our MMKG also includes frame-by-frame fine-grained changes, such as bounding boxes within video frames. In addition, we provide support tools for querying our MMKG. As an application example, we demonstrate that our MMKG facilitates benchmarking vision-language models by providing the necessary vision-language datasets for a tailored task.

Figures (4)

• Figure 1: Illustration of VHAKG
• Figure 2: Number of actions (red means new actions that became executable by this study)
• Figure 3: Example question and query pattern
• Figure 4: Distribution of actions in test datasets