LLM-guided Hierarchical Retrieval

TL;DR

This paper introduces LATTICE, an LLM-guided hierarchical retrieval framework that organizes a large corpus into a semantic tree offline and guides online search with a calibrated, reasoning-enabled LLM traversal. By converting local, noisy relevance judgments into a globally coherent path relevance score, LATTICE achieves state-of-the-art zero-shot performance on the BRIGHT benchmark and remains competitive with fine-tuned baselines on static corpora. The approach combines two tree-construction strategies (bottom-up clustering and top-down divisive summarization) with a lightweight, greedy best-first traversal that scales with logarithmic search complexity. Empirically, LATTICE delivers strong ranking and retrieval metrics (nDCG@10, Recall@100) while enabling robust, update-friendly reasoning over large document collections, albeit with challenges on dynamically changing corpora that may require adaptive summaries.

Abstract

Modern IR systems are increasingly tasked with answering complex, multi-faceted queries that require deep reasoning rather than simple keyword or semantic matching. While LLM-based IR has shown great promise, the prevailing retrieve-then-rerank paradigm inherits the limitations of embedding-based retrieval; parametric generative approaches are difficult to update with new information; and long-context methods that place the entire corpus in context are computationally infeasible for large document collections. To address these challenges, we introduce LATTICE, a hierarchical retrieval framework that enables an LLM to reason over and navigate large corpora with logarithmic search complexity by imposing a semantic tree structure on the corpus. Our approach consists of two stages: (1) an offline phase that organizes the corpus into a semantic hierarchy via either a bottom-up agglomerative strategy or a top-down divisive strategy using multi-level summaries and (2) an online traversal phase where a search LLM navigates this tree. A central challenge in such LLM-guided search is that the model's relevance judgments are noisy, context-dependent, and unaware of the hierarchy, making cross-branch and cross-level comparisons difficult. To overcome this, we propose a traversal algorithm that estimates calibrated latent relevance scores from local LLM outputs and aggregates them into a global path relevance metric. Our training-free framework achieves state-of-the-art zero-shot performance on the reasoning-intensive BRIGHT benchmark, demonstrating up to 9% improvement in Recall@100 and 5% in nDCG@10 over the next best zero-shot baseline. Furthermore, compared to the fine-tuned SOTA method DIVER-v2, LATTICE attains comparable results on BRIGHT subsets that use a static corpus for evaluation.

Figures (10)

• Figure 1: (Left) Recall@100 on BRIGHT’s StackExchange subsets. Comparison between zero-shot LATTICE (Gemini-2.5-flash) against a BM25 retriever (with GPT-4 query expansion) and a fine-tuned dual-encoder ReasonIR-8B Shao2025-qk (with GPT-4 query expansion). LATTICE yields the highest average recall (74.8%) and substantially outperforms BM25 across all subsets (avg +9.5 pp) and ReasonIR-8B on average (+4.0 pp), with particularly large gains on some datasets like Economics and Robotics. (Right) LLM cost (measured in avg. number of input tokens given to LLM) vs. ranking quality (nDCG@10) on the Robotics subset. Reranking baselines (BM25+rerank, ReasonIR-8B+rerank) with varying top-k shortlist use same Gemini-2.5-flash as reranker exhibit early gains but quickly plateau. LATTICE starts with a shallow flat region (cost of traversing tree levels) but then scales more effectively—surpassing the baselines and continuing to improve to a higher final nDCG—demonstrating that guided hierarchical traversal using LLM can be more compute efficient.
• Figure 2: A high-level overview of our proposed framework, LATTICE. The process consists of two stages. (Left) In the offline stage, we organize an unstructured document corpus into a semantic tree. (Right) In the online stage, a search LLM performs a best-first traversal over calibrated path relevance scores to find documents relevant to a user query. The path relevance score is defined as the exponentially moving average of calibrated scores of nodes on the path. Score calibration is achieved by comparing nodes against high-relevance candidates from sibling branches and previously seen leaves, ensuring a globally coherent search.
• Figure 3: An illustration of the search process of LATTICE for a real query from the BRIGHT benchmark. The color of each node corresponds to its computed path relevance; the highlighted yellow path shows the path to ground-truth documents. The search LLM makes a step-by-step decision at each internal node to determine which branch to explore next. The expanded callout provides a "glass box" view into one such decision, detailing the LLM's explicit reasoning process as it scores the child nodes.
• Figure 4: nDCG@10 vs. $\ell$.
• Figure 5: nDCG@10 vs. beam-size.