BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models

TL;DR

Long-context extension for LLMs often trades off quality for cost when relying on chunked retrieval. This paper introduces Landmark Embedding, a chunking-free embedding method that uses landmark tokens (LMK) and a shared encoder to produce contextually informed sentence embeddings from coherent long inputs, enabling effective retrieval with sliding-window processing. It couples a position-aware contrastive objective with a three-stage learning pipeline (distant, weak, and synthetic-data fine-tuning) to leverage abundant and synthetic data for cost-efficient training. Empirical results on LLaMA-2-7B and ChatGPT-3.5-turbo across six long-context benchmarks demonstrate substantial retrieval augmentation gains, especially for models with smaller context windows, highlighting the method's practicality for extending long-context capabilities in real systems.

Abstract

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.

Figures (8)

• Figure 1: Sentence Embedding works with the chunked context, which tends to select the salient sentence. Landmark Embedding maintains a coherent context, which enables it to select the right sentence.
• Figure 2: Architecture for Landmark Embedding. The landmark LMK token is appended to the end of each sentence. A sliding window is employed to handle the input sequence longer than the LLM's context window.
• Figure 3: Weak Supervision (L) and Fine-Tuning (R).
• Figure 4: Length distribution of evaluation data.
• Figure 5: Needle in a haystack test.