Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval

TL;DR

This paper tackles text-video retrieval by addressing the inherent 1-to-N matching challenge and irregular data representations. It introduces TV-ProxyNet, which replaces the single text query with multiple text proxies and uses a director-dash mechanism to control their direction and distance, effectively decomposing into N 1-to-1 relationships. The training objective combines proxy-to-video and standard text-to-video contrastive losses, and inference fuses the proxy-based logits with a weighted term $\gamma$ to enhance discrimination, as in $s(\mathbf{t}_{q_i}, \mathbf{p}_{1_j}) + \gamma s(\mathbf{t}_{p_{i,j}}, \mathbf{p}_{1_j})$. Experiments on MSRVTT, DiDeMo, and ActivityNet Captions show consistent gains, achieving state-of-the-art results on MSRVTT and ActivityNet and improving DiDeMo by about 2.0% in R@1, demonstrating improved semantic mapping and reduced susceptibility to irregularities.

Abstract

Text-video retrieval (TVR) has seen substantial advancements in recent years, fueled by the utilization of pre-trained models and large language models (LLMs). Despite these advancements, achieving accurate matching in TVR remains challenging due to inherent disparities between video and textual modalities and irregularities in data representation. In this paper, we propose Text-Video-ProxyNet (TV-ProxyNet), a novel framework designed to decompose the conventional 1-to-N relationship of TVR into N distinct 1-to-1 relationships. By replacing a single text query with a series of text proxies, TV-ProxyNet not only broadens the query scope but also achieves a more precise expansion. Each text proxy is crafted through a refined iterative process, controlled by mechanisms we term as the director and dash, which regulate the proxy's direction and distance relative to the original text query. This setup not only facilitates more precise semantic alignment but also effectively manages the disparities and noise inherent in multimodal data. Our experiments on three representative video-text retrieval benchmarks, MSRVTT, DiDeMo, and ActivityNet Captions, demonstrate the effectiveness of TV-ProxyNet. The results show an improvement of 2.0% to 3.3% in R@1 over the baseline. TV-ProxyNet achieved state-of-the-art performance on MSRVTT and ActivityNet Captions, and a 2.0% improvement on DiDeMo compared to existing methods, validating our approach's ability to enhance semantic mapping and reduce error propensity.

Figures (6)

• Figure 1: (a): Text annotations being overly simplistic and providing insufficient distinctiveness for video is one of the irregularity problems. (b): Illustration of text proxy. The director leader and the dash mechanism influence the direction and distance of proxies, effectively drawing them closer to positive and further from negative video token clusters. The concept of video proxies is from the CLIP-ViP xue2023clip, and different video proxies can be seen as different video features containing different semantics.
• Figure 2: TV-ProxyNet converts the traditional 1-to-$N$ relationship into $N$ distinct 1-to-1 relationships. Each 1-to-1 relationship is unique, compelling the model to recognize the unique data patterns inherent in each pair, excluding irregular factors.
• Figure 3: Framework of TV-ProxyNet. After obtaining video proxies and text query from backbone encoders, we generate direction leader to guide the direction of director and dash to specify the distance between text query and text proxy. During the training stage, $\mathcal{L}_p$ aims to optimize the mentioned $N$ 1-to-1 relationships while $\mathcal{L}_{pos}$ aims to regulate the positive proxy.
• Figure 4: The R@1 results at different $\gamma$.
• Figure 5: The t-SNE visualization of feature space, the red, blue, and green points represent $\mathbf{q}$, video $\mathbf{p}_1$, and text $\mathbf{t}_q$, respectively.