Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models
Zheng Xing, Weibing Zhao
TL;DR
The paper tackles unsupervised HMS by introducing temporal vision semantics (TVS) learned via large language models (LLMs) and directly integrating this semantic temporal information into a subspace clustering framework. TVS is extracted as temporal neighborhoods through LLM-based judgments on consecutive frames, encoded in a TVS matrix \mathbf{G}, and used to regularize subspace embeddings via the term \|\mathbf{Z}\mathbf{G}\|_{2,1} as well as clustering via a temporally constrained \mathbf{Q}. A tight, ADMM-based, feedback-enabled optimization is developed to jointly update the subspace embedding \mathbf{Z}, the auxiliary variable \mathbf{H}=\mathbf{Z}\mathbf{G}, and the cluster indicators \mathbf{Q}, with provable convergence to a stationary point. Experiments on four HMS benchmarks demonstrate state-of-the-art performance, highlighting the value of semantically guided temporal priors for robust motion segmentation, while ablations and analyses dissect the contributions of TVS modeling, joint optimization, and LLM choice/prompts. The work suggests a practical path to leveraging multimodal reasoning to enhance unsupervised video understanding and segmentation in real-world scenarios.
Abstract
Unsupervised human motion segmentation (HMS) can be effectively achieved using subspace clustering techniques. However, traditional methods overlook the role of temporal semantic exploration in HMS. This paper explores the use of temporal vision semantics (TVS) derived from human motion sequences, leveraging the image-to-text capabilities of a large language model (LLM) to enhance subspace clustering performance. The core idea is to extract textual motion information from consecutive frames via LLM and incorporate this learned information into the subspace clustering framework. The primary challenge lies in learning TVS from human motion sequences using LLM and integrating this information into subspace clustering. To address this, we determine whether consecutive frames depict the same motion by querying the LLM and subsequently learn temporal neighboring information based on its response. We then develop a TVS-integrated subspace clustering approach, incorporating subspace embedding with a temporal regularizer that induces each frame to share similar subspace embeddings with its temporal neighbors. Additionally, segmentation is performed based on subspace embedding with a temporal constraint that induces the grouping of each frame with its temporal neighbors. We also introduce a feedback-enabled framework that continuously optimizes subspace embedding based on the segmentation output. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on four benchmark human motion datasets.