Learning Action Hierarchies via Hybrid Geometric Diffusion

TL;DR

HybridTAS introduces a diffusion-based temporal action segmentation framework that explicitly models action hierarchies by coupling Euclidean label supervision with hyperbolic latent representations. The method proceeds in two phases: a Stabilization Phase that learns hyperbolic action prototypes and a Guidance Phase that steers denoising along geodesic paths toward these prototypes, using four hyperbolic losses alongside standard cross-entropy. By leveraging the Poincaré ball geometry, exponential maps, and hyperbolic distances, the approach captures coarse-to-fine action structure and uncertainty, yielding state-of-the-art results on GTEA, 50Salads, and Breakfast with fewer inference steps. The work demonstrates that geometry-aware denoising improves hierarchical representation, temporal coherence, and boundary precision, offering practical gains for TAS in real-world video understanding tasks.

Abstract

Temporal action segmentation is a critical task in video understanding, where the goal is to assign action labels to each frame in a video. While recent advances leverage iterative refinement-based strategies, they fail to explicitly utilize the hierarchical nature of human actions. In this work, we propose HybridTAS - a novel framework that incorporates a hybrid of Euclidean and hyperbolic geometries into the denoising process of diffusion models to exploit the hierarchical structure of actions. Hyperbolic geometry naturally provides tree-like relationships between embeddings, enabling us to guide the action label denoising process in a coarse-to-fine manner: higher diffusion timesteps are influenced by abstract, high-level action categories (root nodes), while lower timesteps are refined using fine-grained action classes (leaf nodes). Extensive experiments on three benchmark datasets, GTEA, 50Salads, and Breakfast, demonstrate that our method achieves state-of-the-art performance, validating the effectiveness of hyperbolic-guided denoising for the temporal action segmentation task.

Figures (9)

• Figure 1: Embeddings in Euclidean and Hyperbolic Space. (a) In the Euclidean space, embeddings tend to cluster towards the origin, making it difficult to distinguish classes. (b) On the other hand, the hyperbolic space allows embeddings to spread due to its exponential distance growth. Additionally, hyperbolic geometry naturally provides information on hierarchy in data, class boundaries, and uncertainty in predictions HybImgSeg.
• Figure 2: Model Architecture. Our architecture builds upon DiffAct DiffAct, but introduces a hybrid design that operates jointly in Euclidean and hyperbolic spaces to utilize hierarchical action relationships. The Euclidean losses are applied in the label space, acting on the predicted action probabilities, while the hyperbolic losses are computed in the embedding space, directly supervising the outputs from the decoder's final layer.
• Figure 3: Denoising trajectory in hyperbolic space. Our hyperbolic loss functions guide the diffusion model to align its denoising trajectory along the geodesic between the origin and the target action prototype. This enforces the model to follow a hierarchical, coarse-to-fine progression in the label embedding space.
• Figure 4: Euclidean and Hyperbolic UMAP on the GTEA dataset gtea. UMAP projections of action embeddings from DiffAct DiffAct in Euclidean space (left) and HybridTAS (Ours) in hyperbolic space (right). Hyperbolic embeddings yield well-separated clusters, with background states centralized and specific actions arranged radially. Further, it better preserves inter-class boundaries and highlights hierarchical structure across tasks. Refer to Fig. \ref{['fig:cluster_dist']} (Appendix) for cluster centroid specific distances.
• Figure 5: Qualitative results on the GTEA dataset gtea. We present a comparison of segmentation outputs of DiffAct DiffAct and HybridTAS (Ours) on S3_CofHoney_C1 (top) and S3_Cheese_C1 (bottom) with dashed boxes representing areas of improvement. HybridTAS yields clearer temporal boundaries, better preserves short actions, and maintains semantic consistency across transitions.