ELITE: Embedding-Less retrieval with Iterative Text Exploration

TL;DR

The paper addresses the inefficiencies and misalignment of embedding-based retrieval in long-context QA by proposing an embedding-free retrieval framework that leverages LLM reasoning through an iterative, cognitively inspired search process. It decomposes retrieval into Generation, Collection, Evaluation via an importance score, and Iterative Exploration over an implicit knowledge graph, avoiding dense embeddings and explicit graphs. Experiments on NovelQA and Marathon show strong accuracy gains across model scales, with dramatically reduced offline preprocessing, storage, and overall latency compared with traditional RAG baselines. This approach offers real-time, transparent retrieval policies and scalable performance for long-context question answering tasks.

Abstract

Large Language Models (LLMs) have achieved impressive progress in natural language processing, but their limited ability to retain long-term context constrains performance on document-level or multi-turn tasks. Retrieval-Augmented Generation (RAG) mitigates this by retrieving relevant information from an external corpus. However, existing RAG systems often rely on embedding-based retrieval trained on corpus-level semantic similarity, which can lead to retrieving content that is semantically similar in form but misaligned with the question's true intent. Furthermore, recent RAG variants construct graph- or hierarchy-based structures to improve retrieval accuracy, resulting in significant computation and storage overhead. In this paper, we propose an embedding-free retrieval framework. Our method leverages the logical inferencing ability of LLMs in retrieval using iterative search space refinement guided by our novel importance measure and extend our retrieval results with logically related information without explicit graph construction. Experiments on long-context QA benchmarks, including NovelQA and Marathon, show that our approach outperforms strong baselines while reducing storage and runtime by over an order of magnitude.

Figures (2)

• Figure 1: algorithm flowchart
• Figure 2: Breakdown of total time consumption across different stages.