StructAlign: Structured Cross-Modal Alignment for Continual Text-to-Video Retrieval

TL;DR

CTVR must learn from a stream of text–video pairs without forgetting previously learned categories. The authors introduce StructAlign, which enforces a simplex Equiangular Tight Frame (ETF) geometry with C prototypes p_c in the shared embedding space, where p_c^T p_{c'} ≈ -1/(C-1) for c ≠ c' and prototypes anchor each category. They implement two losses, L_ETF for cross-modal ETF alignment and L_CRP for preserving cross-modal similarity relations to constrain intra-modal drift, along with pseudo-feature anchors for old categories. Empirical results on MSRVTT and ACTNET show strong improvements over state-of-the-art continual retrieval methods, validating the ETF prior and the relational regularizer for robust multimodal continual learning.

Abstract

Continual Text-to-Video Retrieval (CTVR) is a challenging multimodal continual learning setting, where models must incrementally learn new semantic categories while maintaining accurate text-video alignment for previously learned ones, thus making it particularly prone to catastrophic forgetting. A key challenge in CTVR is feature drift, which manifests in two forms: intra-modal feature drift caused by continual learning within each modality, and non-cooperative feature drift across modalities that leads to modality misalignment. To mitigate these issues, we propose StructAlign, a structured cross-modal alignment method for CTVR. First, StructAlign introduces a simplex Equiangular Tight Frame (ETF) geometry as a unified geometric prior to mitigate modality misalignment. Building upon this geometric prior, we design a cross-modal ETF alignment loss that aligns text and video features with category-level ETF prototypes, encouraging the learned representations to form an approximate simplex ETF geometry. In addition, to suppress intra-modal feature drift, we design a Cross-modal Relation Preserving loss, which leverages complementary modalities to preserve cross-modal similarity relations, providing stable relational supervision for feature updates. By jointly addressing non-cooperative feature drift across modalities and intra-modal feature drift, StructAlign effectively alleviates catastrophic forgetting in CTVR. Extensive experiments on benchmark datasets demonstrate that our method consistently outperforms state-of-the-art continual retrieval approaches.

Figures (8)

• Figure 1: (a) In continual learning, feature drift refers to the phenomenon where feature representations shift toward new tasks, gradually deviating from feature representations learned for previous tasks. (b) In multimodal continual learning, features from different modalities tend to drift in a non-cooperative manner, which ultimately results in modality misalignment. (c) Our StructAlign adopts a simplex ETF geometry as a unified geometric prior to mitigate modality misalignment during multimodal continual learning.
• Figure 2: The overview of StructAlign. Based on a simplex ETF geometric prior, StructAlign explicitly aligns both real features from the current task and pseudo features from previous tasks with their corresponding ETF prototypes via a cross-modal ETF alignment loss $\mathcal{L}_{ETF}$ (Sec. 4.2). Moreover, a cross-modal relation preserving loss $\mathcal{L}_{\mathrm{CRP}}$ is introduced to stabilize intra-modal feature updates by preserving cross-modal similarity relations under the ETF-induced geometry (Sec. 4.3 and Fig. 3).
• Figure 3: Illustration of the cross-modal relation preserving loss. By leveraging the complementary modality, cross-modal similarity matrices $\mathbf{S}^{k}$ and $\mathbf{S}^{k-1}$ are constructed under the current model $\theta_k$ and the previous model $\theta_{k-1}$, respectively. Treating $\mathbf{S}^{k-1}$ as a fixed relational anchor, the current similarity matrix $\mathbf{S}^{k}$ is optimized to preserve its relational structure. For clarity, the text modality is shown as the auxiliary modality, while the formulation is symmetric across modalities.
• Figure 4: The comparison performance on MSRVTT (a)-(b) and ACTNET (c)-(d).
• Figure 5: Trade-off between retrieval performance and trainable Parameters.