Cost-efficient Knowledge-based Question Answering with Large Language Models

TL;DR

The paper addresses cost-efficient knowledge-based question answering (KBQA) by combining large language models (LLMs) with knowledge-graph-based small models (KGMs). It introduces Coke, a tailored multi-armed bandit that selects the most promising model for each question under a budget, using cluster-level Thompson Sampling, a context-aware arm policy, and a cost-regret constraint. The approach yields a favorable accuracy-cost Pareto frontier, achieving up to 2.74% higher accuracy and up to 20.89% savings in API fees on three domain benchmarks. This has practical impact for deploying KBQA systems in cost-sensitive settings without sacrificing domain-specific reasoning quality.

Abstract

Knowledge-based question answering (KBQA) is widely used in many scenarios that necessitate domain knowledge. Large language models (LLMs) bring opportunities to KBQA, while their costs are significantly higher and absence of domain-specific knowledge during pre-training. We are motivated to combine LLMs and prior small models on knowledge graphs (KGMs) for both inferential accuracy and cost saving. However, it remains challenging since accuracy and cost are not readily combined in the optimization as two distinct metrics. It is also laborious for model selection since different models excel in diverse knowledge. To this end, we propose Coke, a novel cost-efficient strategy for KBQA with LLMs, modeled as a tailored multi-armed bandit problem to minimize calls to LLMs within limited budgets. We first formulate the accuracy expectation with a cluster-level Thompson Sampling for either KGMs or LLMs. A context-aware policy is optimized to further distinguish the expert model subject to the question semantics. The overall decision is bounded by the cost regret according to historical expenditure on failures. Extensive experiments showcase the superior performance of Coke, which moves the Pareto frontier with up to 20.89% saving of GPT-4 fees while achieving a 2.74% higher accuracy on the benchmark datasets.

Figures (5)

• Figure 1: A sketched overview of LLMs and small KGMs in (a). We visualize the Acc./Param size of both pipelines of models in (b). The overlaps among different model predictions are shown in (c).
• Figure 2: A visualization of Pareto frontier of both inferential accuracy and cost saving as budget $\mathcal{B}$ increases on three datasets.
• Figure 3: A 3D toy visualization of the selection regret on three datasets as iteration $k$ goes.
• Figure 4: A case study of the model selection on three domain-specific datasets as $k$ goes. The color changes from deep to shallow indicates an exploration process, while an exploitation reversely.
• Figure 5: Performance changes based on the search of $\lambda$ on three datasets.