GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering

TL;DR

GraphOTTER introduces an explicit graph-based reasoning framework for complex table QA, transforming tables into undirected graphs where both headers and data are nodes. It uses a Thought-Action-Update loop with a small set of intermediate actions to perform step-by-step reasoning, guided by a reasoning trace that records visited nodes. By focusing the reasoning on a graph-subset relevant to the question, GraphOTTER better filters irrelevant information and achieves state-of-the-art results across two benchmarks and two LLM backbones. The approach demonstrates notable gains over baselines and highlights the value of explicit graph reasoning and header inference in handling complex table structures. Future work includes extending to multi-turn scenarios and exploring more advanced graph actions to further improve efficiency and accuracy.

Abstract

Complex Table Question Answering involves providing accurate answers to specific questions based on intricate tables that exhibit complex layouts and flexible header locations. Despite considerable progress having been made in the LLM era, the reasoning processes of existing methods are often implicit, feeding the entire table into prompts, making it difficult to effectively filter out irrelevant information in the table. To this end, we propose GraphOTTER that explicitly establishes the reasoning process to pinpoint the correct answers. In particular, GraphOTTER leverages a graph-based representation, transforming the complex table into an undirected graph. It then conducts step-by-step reasoning on the graph, with each step guided by a set of pre-defined intermediate reasoning actions. As such, it constructs a clear reasoning path and effectively identifies the answer to a given question. Comprehensive experiments on two benchmark datasets and two LLM backbones demonstrate the effectiveness of GraphOTTER. Further analysis indicates that its success may be attributed to the ability to efficiently filter out irrelevant information, thereby focusing the reasoning process on the most pertinent data. Our code and experimental datasets are available at \url{https://github.com/JDing0521/GraphOTTER}.

Figures (15)

• Figure 1: Example of complex table QA. Header cells are styled with a dashed background, while the reasoning process is highlighted using color.
• Figure 2: Overview of GraphOTTER. It first transforms the table into a graph representation (For clarity, this graph only shows some of the edges between nodes in the same row/column). Then, it establishes a step-by-step reasoning process on this graph, guided by a reasoning trace and a set of pre-defined intermediate actions (Note that the nodes in the reasoning trace are colored). These actions facilitate navigation and reasoning within the graph structure, leading to a more explicit and efficient approach for complex table QA.
• Figure 3: Average percentage of cells/nodes required for generating the final answer. We report results on explicit reasoning methods, while for implicit reasoning based methods, they use all the cells in the whole table.
• Figure 4: Illustration of the iteration efficiency of different methods. GraphOTTER achieves superior performance while requiring fewer iterations.
• Figure 5: Impact of reasoning trace initialization (i.e., tuning top-k relevant cells to initialize the trace). Proper reasoning trace initialization ensures that GraphOTTER begins its reasoning with enough information without being overwhelmed by irrelevant details.