ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models

TL;DR

This work critiques traditional embedding benchmarks for overlooking advanced NLP capabilities such as factuality, safety, and instruction-following. It introduces ATEB, a benchmark that reframes diverse datasets as retrieval tasks via label augmentation with explanations, enabling symmetric dual encoders to tackle complex tasks without architectural changes. The authors demonstrate substantial improvements on factuality and safety tasks through single-task fine-tuning and show that adapter-based fine-tuning offers a resource-efficient alternative. Overall, ATEB exposes gaps in current embeddings while providing a practical pathway to elevate their performance on real-world, multi-faceted NLP tasks.

Abstract

Traditional text embedding benchmarks primarily evaluate embedding models' capabilities to capture semantic similarity. However, more advanced NLP tasks require a deeper understanding of text, such as safety and factuality. These tasks demand an ability to comprehend and process complex information, often involving the handling of sensitive content, or the verification of factual statements against reliable sources. We introduce a new benchmark designed to assess and highlight the limitations of embedding models trained on existing information retrieval data mixtures on advanced capabilities, which include factuality, safety, instruction following, reasoning and document-level understanding. This benchmark includes a diverse set of tasks that simulate real-world scenarios where these capabilities are critical and leads to identification of the gaps of the currently advanced embedding models. Furthermore, we propose a novel method that reformulates these various tasks as retrieval tasks. By framing tasks like safety or factuality classification as retrieval problems, we leverage the strengths of retrieval models in capturing semantic relationships while also pushing them to develop a deeper understanding of context and content. Using this approach with single-task fine-tuning, we achieved performance gains of 8\% on factuality classification and 13\% on safety classification. Our code and data will be publicly available.