TreeRare: Syntax Tree-Guided Retrieval and Reasoning for Knowledge-Intensive Question Answering

TL;DR

TreeRare tackles knowledge-intensive QA by interleaving syntax-driven question decomposition with bottom-up retrieval and reasoning. It uses a syntax tree to guide subcomponent query generation, targeted retrieval, and subcomponent answering, followed by aggregating evidence for a final answer. Across five benchmarks and multiple LLM backbones, TreeRare yields substantial improvements over state-of-the-art baselines, and the ablation studies confirm the critical role of syntax-guided retrieval and per-node reasoning. The approach reduces hallucinations and ambiguity by grounding evidence at fine-grained sub-phrases, though it introduces higher computational costs and depends on parsing quality.

Abstract

In real practice, questions are typically complex and knowledge-intensive, requiring Large Language Models (LLMs) to recognize the multifaceted nature of the question and reason across multiple information sources. Iterative and adaptive retrieval, where LLMs decide when and what to retrieve based on their reasoning, has been shown to be a promising approach to resolve complex, knowledge-intensive questions. However, the performance of such retrieval frameworks is limited by the accumulation of reasoning errors and misaligned retrieval results. To overcome these limitations, we propose TreeRare (Syntax Tree-Guided Retrieval and Reasoning), a framework that utilizes syntax trees to guide information retrieval and reasoning for question answering. Following the principle of compositionality, TreeRare traverses the syntax tree in a bottom-up fashion, and in each node, it generates subcomponent-based queries and retrieves relevant passages to resolve localized uncertainty. A subcomponent question answering module then synthesizes these passages into concise, context-aware evidence. Finally, TreeRare aggregates the evidence across the tree to form a final answer. Experiments across five question answering datasets involving ambiguous or multi-hop reasoning demonstrate that TreeRare achieves substantial improvements over existing state-of-the-art methods.

Figures (16)

• Figure 1: Comparison of and ReActyao2022react on a multihop question. decomposes the question into structured sub-questions and retrieves focused evidence. In contrast, ReAct fails to generate useful query and misidentifies the region.
• Figure 2: Overview of the framework. Given a knowledge-intensive question, first decomposes it into a syntax tree. It then traverses the tree in a bottom-up manner. At each node, the framework generates subcomponent-based queries conditioned on evidence from its child nodes and the current phrase. These queries guide document retrieval, and a subcomponent question answering module extracts evidence related to the generated queries. Finally, all node-level evidence is aggregated to produce the final answer.
• Figure 3: Distribution of outcome types for ReAct and on randomly sampled multihop QA.
• Figure 4: Total GPT-4o-mini API cost for (CT), (DT), TOT, ReAct, SC, and COT across HotpotQA, MuSiQue, 2WikiMQA, AmbigDoc, and ASQA
• Figure 5: Total GPT-4o-mini API cost (input + output tokens) for (CT), (DT), TOT, ReAct, SC, and COT across HotpotQA, MuSiQue, 2WikiMQA, AmbigDoc, and ASQA based on OpenAI pricing.