HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling

TL;DR

HieraMamba tackles long-video temporal grounding by learning a hierarchical, multi-scale representation through Anchor-MambaPooling blocks that compress video content into a pyramid of anchors while preserving both global context and local detail. Two contrastive losses, ACC and SPC, enforce structural coherence across scales and semantic alignment with ground-truth segments, enabling precise and temporally faithful localization in hour-long videos. The approach achieves state-of-the-art results on Ego4D-NLQ, MAD, and TACoS, while maintaining linear-time inference via Mamba-based selective scanning. This combination of hierarchical token compression and contrastive supervision yields accurate grounding with scalable efficiency, offering a general framework for long-form video understanding and potential extensions to related tasks.

Abstract

Video temporal grounding, the task of localizing the start and end times of a natural language query in untrimmed video, requires capturing both global context and fine-grained temporal detail. This challenge is particularly pronounced in long videos, where existing methods often compromise temporal fidelity by over-downsampling or relying on fixed windows. We present HieraMamba, a hierarchical architecture that preserves temporal structure and semantic richness across scales. At its core are Anchor-MambaPooling (AMP) blocks, which utilize Mamba's selective scanning to produce compact anchor tokens that summarize video content at multiple granularities. Two complementary objectives, anchor-conditioned and segment-pooled contrastive losses, encourage anchors to retain local detail while remaining globally discriminative. HieraMamba sets a new state-of-the-art on Ego4D-NLQ, MAD, and TACoS, demonstrating precise, temporally faithful localization in long, untrimmed videos.

Figures (7)

• Figure 1: HieraMamba enables hierarchical, linear-time temporal grounding in long untrimmed videos. Top row: a cooking clip and two queries—Q1 spans a long interval, Q2 a very short one. Middle left: uniform down-sampling (gray squares) drops frames and loses evidence for the queries. Bottom left: fixed sliding windows split segments at window boundaries but are susceptible to fragmenting (red dashed lines). Right: HieraMamba builds on our stacked Anchor-MambaPooling (AMP) blocks to construct a multi-scale temporal hierarchy for precise, query-specific localization across levels. For example, the brief 'stove on' moment in Q2 is captured by fine-scale embeddings in the first layer, while Q1's broader context ('prepping ingredients') is naturally represented by the longer, coarser embeddings at the top layer.
• Figure 2: Overview of the HieraMamba Architecture.(a) Frozen backbones extract video clip and text token features. The hierarchical video encoder, a stack of $L$ AMP blocks, builds a multi-scale pyramid $\mathcal{V}_{\text{pyr}}$, which is fused with text features and decoded to predict timestamps. (b) Each AMP block receives anchors from the previous layer ($A^{(l)}$), interleaves them with new compressed anchors ($A^{(l+1)}$), applies a bidirectional Mamba scan for global context, and refines local details. The block outputs refined tokens ($\tilde{V}^{(l)}$) and downsampled anchors ($A^{(l+1)}$) fed to the next block. Repeating this $L$ times and collecting the refined outputs $\{\tilde{V}^{(l)}\}_{l=0}^{L-1}$ forms the multi-scale hierarchy $\mathcal{V}_{\text{pyr}}$. (c) Two contrastive losses guide training. The self-supervised ACC loss enforces hierarchy consistency by pulling anchors toward their constituent frames and pushing from distant anchors. The supervised SPC loss provides semantic alignment between ground-truth segments and surrounding context. Together, they yield compact, distinctive, and query-aligned anchors.
• Figure 3: Qualitative Visualization. Qualitative visualization of queries, ground truth, and our predictions. A single video can contain queries that require grounding short, medium, or long temporal spans, necessitating flexible reasoning at different scales. HieraMamba, with its rich multi-scale semantics, effectively adapts to these varying granularities.
• Figure 4: Accuracy-Compute Trade-off. We plot average recall on the MAD-v2 eval set against computational cost (FLOPs), with FLOPs measured for a single forward pass on a sequence simulating the $\sim100$ minute average video duration. HieraMamba achieves state-of-the-art accuracy with significantly lower computational cost than strong baselines.
• Figure 5: Qualitative Results Comparison with SnAG snag.