Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs

Md Adnan Arefeen, Biplob Debnath, Srimat Chakradhar

TL;DR

LeanContext tackles the high cost of domain-specific QA with LLMs by performing query-aware context reduction. It uses reinforcement learning to adaptively select a top-$k$ set of sentences and reduces the remaining content with an open-source text-reduction method, forming a compact, ordered context for the LLM. Across ArXiv and BBC News datasets, it achieves $37.29\%$ to $67.81\%$ cost savings with only $1.41\%$ to $2.65\%$ ROUGE degradation, and further gains ($13.22\%$ to $24.61\%$) when combined with free summarizers. The approach demonstrates a practical, cost-effective pathway for domain-specific QA using pay-per-use LLMs without substantial accuracy loss.

Abstract

Question-answering (QA) is a significant application of Large Language Models (LLMs), shaping chatbot capabilities across healthcare, education, and customer service. However, widespread LLM integration presents a challenge for small businesses due to the high expenses of LLM API usage. Costs rise rapidly when domain-specific data (context) is used alongside queries for accurate domain-specific LLM responses. One option is to summarize the context by using LLMs and reduce the context. However, this can also filter out useful information that is necessary to answer some domain-specific queries. In this paper, we shift from human-oriented summarizers to AI model-friendly summaries. Our approach, LeanContext, efficiently extracts $k$ key sentences from the context that are closely aligned with the query. The choice of $k$ is neither static nor random; we introduce a reinforcement learning technique that dynamically determines $k$ based on the query and context. The rest of the less important sentences are reduced using a free open source text reduction method. We evaluate LeanContext against several recent query-aware and query-unaware context reduction approaches on prominent datasets (arxiv papers and BBC news articles). Despite cost reductions of $37.29\%$ to $67.81\%$, LeanContext's ROUGE-1 score decreases only by $1.41\%$ to $2.65\%$ compared to a baseline that retains the entire context (no summarization). Additionally, if free pretrained LLM-based summarizers are used to reduce context (into human consumable summaries), LeanContext can further modify the reduced context to enhance the accuracy (ROUGE-1 score) by $13.22\%$ to $24.61\%$.

LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs

TL;DR

LeanContext tackles the high cost of domain-specific QA with LLMs by performing query-aware context reduction. It uses reinforcement learning to adaptively select a top- set of sentences and reduces the remaining content with an open-source text-reduction method, forming a compact, ordered context for the LLM. Across ArXiv and BBC News datasets, it achieves to cost savings with only to ROUGE degradation, and further gains ( to ) when combined with free summarizers. The approach demonstrates a practical, cost-effective pathway for domain-specific QA using pay-per-use LLMs without substantial accuracy loss.

Abstract

Question-answering (QA) is a significant application of Large Language Models (LLMs), shaping chatbot capabilities across healthcare, education, and customer service. However, widespread LLM integration presents a challenge for small businesses due to the high expenses of LLM API usage. Costs rise rapidly when domain-specific data (context) is used alongside queries for accurate domain-specific LLM responses. One option is to summarize the context by using LLMs and reduce the context. However, this can also filter out useful information that is necessary to answer some domain-specific queries. In this paper, we shift from human-oriented summarizers to AI model-friendly summaries. Our approach, LeanContext, efficiently extracts key sentences from the context that are closely aligned with the query. The choice of is neither static nor random; we introduce a reinforcement learning technique that dynamically determines based on the query and context. The rest of the less important sentences are reduced using a free open source text reduction method. We evaluate LeanContext against several recent query-aware and query-unaware context reduction approaches on prominent datasets (arxiv papers and BBC news articles). Despite cost reductions of to , LeanContext's ROUGE-1 score decreases only by to compared to a baseline that retains the entire context (no summarization). Additionally, if free pretrained LLM-based summarizers are used to reduce context (into human consumable summaries), LeanContext can further modify the reduced context to enhance the accuracy (ROUGE-1 score) by to .
Paper Structure (23 sections, 5 equations, 5 figures, 5 tables, 2 algorithms)

This paper contains 23 sections, 5 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Domain-specific QA System
  4. (a) Domain data ingestion:
  5. (b) QA:
  6. LeanContext
  7. Adaptive-$k$:
  8. Details of RL:
  9. State:
  10. Action:
  11. Reward:
  12. Training algorithm:
  13. LeanContext Inference
  14. Experimental Settings
  15. Dataset:
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Compared to the original context LeanContext only drops in $\sim$2% ROUGE-1 score with $\sim$ 68% savings on BBCNews dataset li2023unlocking.
  • Figure 2: Workflow of a domain-specific QA system.
  • Figure 3: LeanContext System
  • Figure 4: Illustration of the reduction of less important sentences while keeping the top-k sentences intact in a context.
  • Figure 5: Adaptive-$k$ ratio selected by the BBCNews RL agent based on queries.