From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding

TL;DR

The paper tackles semantic fragmentation in human action understanding by introducing a structured, VerbNet-based semantic space and unifying 28 multi-modal datasets into Pangea. It presents P2S, a cross-modal mapping from physical representations to semantic verb nodes, using semantic disentanglement and Lorentz-hyperbolic encoding to capture hierarchical relationships. A complementary S2P component enables generation of 3D motions from semantic conditions, and extensive experiments demonstrate strong transfer learning performance across image, video, and 3D benchmarks. The work provides a scalable, interpretable framework for open-action understanding with broad implications for cross-domain learning and future joint P2S/S2P modeling.

Abstract

Action understanding has attracted long-term attention. It can be formed as the mapping from the physical space to the semantic space. Typically, researchers built datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Datasets are incompatible with each other like "Isolated Islands" due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that we need a more principled semantic space to concentrate the community efforts and use all datasets together to pursue generalizable action learning. To this end, we design a structured action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly, we propose a novel model mapping from the physical space to semantic space to fully use Pangea. In extensive experiments, our new system shows significant superiority, especially in transfer learning. Our code and data will be made public at https://mvig-rhos.com/pangea.

Figures (21)

• Figure 1: "Isolated islands". The semantic gap brings a great challenge to general action understanding.
• Figure 2: Verb tree. The conventional action semantics (e.g., hold, hug) can be mapped into node semantics (e.g., touch-20-1, support-15.3). The proposed semantic space has abundant semantic and geometric knowledge.
• Figure 2: Results on the image benchmark HICO hico.
• Figure 3: Gathered datasets in Pangea.
• Figure 4: P2S mapping. Given a sample, we obtain its representation $V$ via encoders. $V$ is then aligned with node representations $G$ under the supervision of $Y$. $v_{raw}$ to $V$ is omitted for clarity.