HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

TL;DR

The paper addresses the problem of training self-supervised models on noisy image–text data by moving beyond alignment-focused filtering to also measure specificity. It introduces HYPerbolic Entailment Filtering (HYPE), which combines uni-modal specificity in hyperbolic space with cross-modal CLIP signals via a MERU-based hyperbolic CLIP and entailment-cone concepts, yielding a composite HYPE score that effectively filters underspecified or misaligned samples. The key contributions include defining image and text specificity ($\\epsilon_i$, $\\epsilon_t$), leveraging the negative Lorentzian distance ($-d_{\\mathcal{L}}$) and cosine similarity ($\\cos(\\theta)$), and formulating the HYPE score with $\\text{HYPE}_{\\text{score}} = \\epsilon_i + \\epsilon_t - d_{\\mathcal{L}} + \\cos(\\theta) + c_{IN}$. Empirical results on the DataComp benchmark show state-of-the-art filtering performance for small/medium datasets and demonstrate data-efficient improvements for image-only SSL; the method also enables high-quality image-only data induction and improves retrieval tasks. The work advances data filtering for self-supervised learning by integrating modality-specific signals with hyperbolic geometry, offering practical gains in data quality and SSL efficiency.

Abstract

In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity $ε_{i}$ can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score.

Figures (19)

• Figure 1: Example of HYPE filtering on the Datacomp small pool datacomp. HYPE leverages both uni-modal specificity (text specificity $\epsilon_{t}$ and image specificity $\epsilon_{i}$) and cross-modal similarity (CLIP similarity $cos(\theta)$ as in this figure or negative Lorentzian distance $-d_{\mathcal{L}}$ can be used instead) to effectively identify and eliminate misalignment and underspecification issues on noisy image-text pairs. Figures (a-c) show instances where image-text pairs exhibit high alignment yet are flagged for exclusion due to insufficient specificity: (b) demonstrates low image specificity $\epsilon_{i}$, (c) illustrates low text specificity $\epsilon_{t}$, and (a) indicates low specificity in both aspects. Conversely, Figure (d) presents a scenario with high $\epsilon_{i}$ and $\epsilon_{t}$ but low $cos(\theta)$, highlighting a misalignment between the image and text, evidenced by the absence of an "elephant print".
• Figure 2: Conceptual comparisons of Euclidean embeddings and hyperbolic embeddings.
• Figure 3: We show examples of low and high $\epsilon_{i}$ and $\epsilon_{t}$ from the 12.8M Datacomp small pool, where each percentile group spanned with 20% intervals. Here, a higher value denotes that the instance is more specific (see \ref{['sec:econe']} for details of $\epsilon_{i}$ and $\epsilon_{t}$). The range absolute value and their percentile $p(\cdot)$ of $\epsilon_{i}$ and $\epsilon_{t}$ are also shown. For texts, the lowest $\epsilon_{t}$ texts are empty sentences or the least specific texts that could fit any image, such as "Picture", while the higher $\epsilon_{t}$ texts are generally longer sentences that describe some object in detail. For images, images with low $\epsilon_{i}$ are either background images with no objects or with too many objects, while images with higher $\epsilon_{i}$ are so-called iconic images, which contain a single object that can be described with precision.
• Figure 4: Visual example of aper \ref{['eq:aper']}, ext \ref{['eq:ext']} and entailment loss \ref{['eq:le']}.
• Figure 5: Comparisons with baseline filtering methods and HYPE. We show the subsampled Datacomp training set from 10% to 40% and evaluate them across four Datacomp benchmark task groups. Each model was trained four times with varied seeds. 10% and 30% results are the same as \ref{['tab:datacomp']}.