Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval

TL;DR

This work tackles optimization tension and false negatives in text–video retrieval by introducing GARE, which learns pair-specific increments $\Delta_{ij}$ to redistribute gradient flow away from brittle anchor updates. The core mechanism combines a multivariate Taylor expansion of the InfoNCE loss under a trust-region with a lightweight cross-attention module $\psi$ conditioned on the semantic gap, and is stabilized by a relaxed variational information bottleneck. Key contributions include a gradient-structure analysis that motivates pairwise increments, norm- and direction-based regularizers to promote structured updates, and extensive experiments on MSR-VTT, DiDeMo, MSVD, and ActivityNet Captions showing consistent improvements over state-of-the-art methods. The approach yields semantically meaningful, geometrically structured increments that broaden the effective optimization region, improving alignment robustness while maintaining computational efficiency suitable for large-scale retrieval pipelines.

Abstract

Recent progress in text-video retrieval has been largely driven by contrastive learning. However, existing methods often overlook the effect of the modality gap, which causes anchor representations to undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity. Moreover, noisy hard negatives further distort the semantics of anchors. To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $Δ_{ij}$ between text $t_i$ and video $v_j$, redistributing gradients to relieve optimization tension and absorb noise. We derive $Δ_{ij}$ via a multivariate first-order Taylor expansion of the InfoNCE loss under a trust-region constraint, showing that it guides updates along locally consistent descent directions. A lightweight neural module conditioned on the semantic gap couples increments across batches for structure-aware correction. Furthermore, we regularize $Δ$ through a variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation. Code is available at https://github.com/musicman217/GARE-text-video-retrieval.

Figures (11)

• Figure 1: Tension and false-negative challenge vs. our offloading strategy. (a) Owing to the modality gap liang2022mind, gradients from negative samples overlap with the positive direction, creating optimization tension around the anchor $t_i$ and limiting its update freedom. (b) GARE offloads part of this optimization pressure from $t_i$ to learnable increments $\Delta_{ij}$, relaxing the gradient field and absorbing false-negative noise. Each $\Delta_{ij}$ encodes a semantically meaningful correction of the text–video gap.
• Figure 2: Mean and variance of summed gradient (top) and negative gradients (bottom) across 512 dimensions, showing collinear but opposite forces that largely cancel out.
• Figure 3: Qualitative analysis on the MSR-VTT 1k-A validation set. $t_{\text{delta}}$ denotes $t_{\Delta}$. Our method induces greater angular separation between positive pairs (a), redistributes $t_{\Delta}$ norms to release gradient tension (b, c), and pushes $t_{\Delta}$ outward from $v_j$ (d), promoting uniformity.
• Figure 4: Mean and variance of per-dimension gradients, indicating the positive gradients (top) acting on $t_{\Delta_{ii}}$ and $\Delta_{ii}$ and the sum of all negative gradients (bottom) for $t_{\Delta_{ij}}$ and $\Delta_{ij}$.
• Figure 5: R@1 score with varying $\lambda$ for $\Delta$ norm regularization.