HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies

TL;DR

HiddenTables introduces a privacy-preserving cooperative framework where a Solver generates Python code guided by an Oracle’s RISQ prompts, enabling table question answering without exposing table data. The approach yields substantial token efficiency and scalable evaluation across large taxonomies via the PyQTax dataset (116,671 Q-T-A-Python triplets), though it trades off some accuracy relative to encoder-based methods. Empirical analyses across WikiSQL, WikiTQ, and SQA show that early conversational rounds offer the most performance gains, while larger tables and complex operations worsen results, highlighting both the practicality and limitations of code-based, privacy-centric TableQA in real-world deployments. Overall, the work demonstrates a promising path toward secure, scalable LLM-based QA on heterogeneous tabular data and lays groundwork for future improvements in privacy-compliant, industrial-grade table reasoning.

Abstract

A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-3.5-turbo. We propose a cooperative game dubbed "HiddenTables" as a potential resolution to this challenge. In essence, "HiddenTables" is played between the code-generating LLM "Solver" and the "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of "HiddenTables" to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns & labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.

Figures (2)

• Figure 1: Overview of our system apparatus to encourage HiddenTables. The setup requires two agents, an Oracle and the Solver, which may or may not be on the same device. For our purposes, the Solver is a gpt-3.5-turbo LLM agent that handles generation off-site, and therefore potentially offers risk of adversarial attacks. We outline the conversation between our agents, which is a message-passing channel that transfers solution code along with follow-up questions, without exposing any information from the datalake. Finally, the Oracle will provide the answer to the user.
• Figure 2: Outline of our Role, Instructions, Schema, and Question (RISQ) prompt template that the Oracle generates for the Solver. Each instruction was curated to align the Solver's code to work with our tables. For instance, all string comparisons are case insensitive and Unicode normalized. For each prompt component we outline the token complexity, which is bounded by the number of columns $O(c)$ in the schema.