Accurate Table Question Answering with Accessible LLMs

TL;DR

Orchestra presents a multi-agent framework for table question answering (TQA) that enables open-weight LLMs to rival or surpass larger proprietary models by decomposing reasoning into a logic path and a data-access path, coordinated through a memory-enabled two-agent system and a decision agent for final answers. The approach is grounded in a Monte Carlo objective realignment to align with the original TQA goal, and validated across WikiTQ, TabFact, and TableBench with diverse open-weight LLMs, achieving strong improvements over ReAcTable and surpassing GPT-4 baselines with larger open models. Empirical results show substantial accuracy gains, albeit with higher latency, justified by the practical value of accuracy in offline data analysis settings. The work also includes ablations confirming the benefits of context refinement and objective calibration, and discusses limitations such as longer inference times and single-table focus, proposing future work toward multi-table TQA and efficiency enhancements.

Abstract

Given a table T in a database and a question Q in natural language, the table question answering (TQA) task aims to return an accurate answer to Q based on the content of T. Recent state-of-the-art solutions leverage large language models (LLMs) to obtain high-quality answers. However, most rely on proprietary, large-scale LLMs with costly API access, posing a significant financial barrier. This paper instead focuses on TQA with smaller, open-weight LLMs that can run on a desktop or laptop. This setting is challenging, as such LLMs typically have weaker capabilities than large proprietary models, leading to substantial performance degradation with existing methods. We observe that a key reason for this degradation is that prior approaches often require the LLM to solve a highly sophisticated task using long, complex prompts, which exceed the capabilities of small open-weight LLMs. Motivated by this observation, we present Orchestra, a multi-agent approach that unlocks the potential of accessible LLMs for high-quality, cost-effective TQA. Orchestra coordinates a group of LLM agents, each responsible for a relatively simple task, through a structured, layered workflow to solve complex TQA problems -- akin to an orchestra. By reducing the prompt complexity faced by each agent, Orchestra significantly improves output reliability. We implement Orchestra on top of AgentScope, an open-source multi-agent framework, and evaluate it on multiple TQA benchmarks using a wide range of open-weight LLMs. Experimental results show that Orchestra achieves strong performance even with small- to medium-sized models. For example, with Qwen2.5-14B, Orchestra reaches 72.1% accuracy on WikiTQ, approaching the best prior result of 75.3% achieved with GPT-4; with larger Qwen, Llama, or DeepSeek models, Orchestra outperforms all prior methods and establishes new state-of-the-art results across all benchmarks.

Figures (7)

• Figure 1: Running example of answering a TQA query through multiple rounds of LLM reasoning, database interactions via SQL, and postprocessing with Python. Few-shot prompts used for in-context learning (see Section \ref{['subsec:cot_react']} for details) are omitted for clarity.
• Figure 2: Example interactions of the logic and query agents.
• Figure 3: Ablation study of the Monte Carlo method.
• Figure 4: Varying number of interaction rounds.
• Figure 5: Ablation study of reasoning context. The results of ReAcTable paired with Mistral-7B and Llama-8B are truncated by the y-axis.