Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Halo: Domain-Aware Query Optimization for Long-Context Question Answering

Pramod Chunduri, Francisco Romero, Ali Payani, Kexin Rong, Joy Arulraj

Abstract

Long-context question answering (QA) over lengthy documents is critical for applications such as financial analysis, legal review, and scientific research. Current approaches, such as processing entire documents via a single LLM call or retrieving relevant chunks via RAG have two drawbacks: First, as context size increases, response quality can degrade, impacting accuracy. Second, iteratively processing hundreds of input documents can incur prohibitively high costs in API calls. To improve response quality and reduce the number of iterations needed to get the desired response, users tend to add domain knowledge to their prompts. However, existing systems fail to systematically capture and use this knowledge to guide query processing. Domain knowledge is treated as prompt tokens alongside the document: the LLM may or may not follow it, there is no reduction in computational cost, and when outputs are incorrect, users must manually iterate. We present Halo, a long-context QA framework that automatically extracts domain knowledge from user prompts and applies it as executable operators across a multi-stage query execution pipeline. Halo identifies three common forms of domain knowledge - where in the document to look, what content to ignore, and how to verify the answer - and applies each at the pipeline stage where it is most effective: pruning the document before chunk selection, filtering irrelevant chunks before inference, and ranking candidate responses after generation. To handle imprecise or invalid domain knowledge, Halo includes a fallback mechanism that detects low-quality operators at runtime and selectively disables them. Our evaluation across finance, literature, and scientific datasets shows that Halo achieves up to 13% higher accuracy and 4.8x lower cost compared to baselines, and enables a lightweight open-source model to approach frontier LLM accuracy at 78x lower cost.

Halo: Domain-Aware Query Optimization for Long-Context Question Answering

Abstract

Long-context question answering (QA) over lengthy documents is critical for applications such as financial analysis, legal review, and scientific research. Current approaches, such as processing entire documents via a single LLM call or retrieving relevant chunks via RAG have two drawbacks: First, as context size increases, response quality can degrade, impacting accuracy. Second, iteratively processing hundreds of input documents can incur prohibitively high costs in API calls. To improve response quality and reduce the number of iterations needed to get the desired response, users tend to add domain knowledge to their prompts. However, existing systems fail to systematically capture and use this knowledge to guide query processing. Domain knowledge is treated as prompt tokens alongside the document: the LLM may or may not follow it, there is no reduction in computational cost, and when outputs are incorrect, users must manually iterate. We present Halo, a long-context QA framework that automatically extracts domain knowledge from user prompts and applies it as executable operators across a multi-stage query execution pipeline. Halo identifies three common forms of domain knowledge - where in the document to look, what content to ignore, and how to verify the answer - and applies each at the pipeline stage where it is most effective: pruning the document before chunk selection, filtering irrelevant chunks before inference, and ranking candidate responses after generation. To handle imprecise or invalid domain knowledge, Halo includes a fallback mechanism that detects low-quality operators at runtime and selectively disables them. Our evaluation across finance, literature, and scientific datasets shows that Halo achieves up to 13% higher accuracy and 4.8x lower cost compared to baselines, and enables a lightweight open-source model to approach frontier LLM accuracy at 78x lower cost.
Paper Structure (33 sections, 15 equations, 6 figures, 9 tables)

This paper contains 33 sections, 15 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. System Overview
  4. End-to-end workflow
  5. Example Workflow
  6. Extracting Domain Knowledge
  7. Where to focus - Structural directives ($\mathcal{S}$)
  8. What to ignore - Filter directives ($\mathcal{F}$)
  9. How to verify - Validation directives ($\mathcal{V}$)
  10. Applying Domain Knowledge
  11. Directive-driven Query Execution
  12. Optimizing Context Selection
  13. Optimizing Inference
  14. Mitigating Hallucinations
  15. Extensions
  16. ...and 18 more sections

Figures (6)

  • Figure 1: VanillaLLM and RAG cannot utilize user's domain knowledge effectively for long-context QA. Halo extracts this knowledge and systematically applies it in a multi-stage query optimization pipeline.
  • Figure 2: Workflow of Halo: (1) the KnowledgeParser extracts domain directives from the user prompt, (2) the ContextSelector prunes structural elements using the Structural operator and identifies the top-$k$ relevant chunks, (3) the InferenceEngine applies the Filter operator to discard irrelevant chunks and generates candidate responses on the remaining chunks, (4) the Verifier re-ranks the responses using Validate operator--demoting hallucinatory responses.
  • Figure 3: LOTUS program for extracting EPS data from financial filings. The user manually specifies the filter predicate and extraction schema in Python.
  • Figure 4: Structure-aware document preprocessing
  • Figure 5: Applying Filter Operator
  • ...and 1 more figures