Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seek and Solve Reasoning for Table Question Answering

Ruya Jiang, Chun Wang, Weihong Deng

TL;DR

The paper tackles the challenge of complex table-based question answering (TQA) by showing that the reasoning conducted during task simplification can be more valuable than the simplification itself. It introduces the Seek-and-Solve pipeline, which integrates an information-seeking reasoning stage (Seek-CoT) with the final solving step, and proposes a compact single-step TQA-solving prompt distilled from this process using In-Context Learning with SS-CoT demonstrations. Empirical results on HiTab and WikiTableQuestions demonstrate that Seek-CoT improves robustness and accuracy, and that the single-step SS-CoT prompt can match multistep approaches in performance while offering efficiency for real-time use. The work highlights the importance of eliciting structured reasoning in LLMs for complex TQA and suggests broader implications for tasks where reasoning in the prior stage is pivotal.

Abstract

The complexities of table structures and question logic make table-based question answering (TQA) tasks challenging for Large Language Models (LLMs), often requiring task simplification before solving. This paper reveals that the reasoning process during task simplification may be more valuable than the simplified tasks themselves and aims to improve TQA performance by leveraging LLMs' reasoning capabilities. We propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions, integrating these two stages at the reasoning level into a coherent Seek-and-Solve Chain of Thought (SS-CoT). Additionally, we distill a single-step TQA-solving prompt from this pipeline, using demonstrations with SS-CoT paths to guide the LLM in solving complex TQA tasks under In-Context Learning settings. Our experiments show that our approaches result in improved performance and reliability while being efficient. Our findings emphasize the importance of eliciting LLMs' reasoning capabilities to handle complex TQA tasks effectively.

Seek and Solve Reasoning for Table Question Answering

TL;DR

The paper tackles the challenge of complex table-based question answering (TQA) by showing that the reasoning conducted during task simplification can be more valuable than the simplification itself. It introduces the Seek-and-Solve pipeline, which integrates an information-seeking reasoning stage (Seek-CoT) with the final solving step, and proposes a compact single-step TQA-solving prompt distilled from this process using In-Context Learning with SS-CoT demonstrations. Empirical results on HiTab and WikiTableQuestions demonstrate that Seek-CoT improves robustness and accuracy, and that the single-step SS-CoT prompt can match multistep approaches in performance while offering efficiency for real-time use. The work highlights the importance of eliciting structured reasoning in LLMs for complex TQA and suggests broader implications for tasks where reasoning in the prior stage is pivotal.

Abstract

The complexities of table structures and question logic make table-based question answering (TQA) tasks challenging for Large Language Models (LLMs), often requiring task simplification before solving. This paper reveals that the reasoning process during task simplification may be more valuable than the simplified tasks themselves and aims to improve TQA performance by leveraging LLMs' reasoning capabilities. We propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions, integrating these two stages at the reasoning level into a coherent Seek-and-Solve Chain of Thought (SS-CoT). Additionally, we distill a single-step TQA-solving prompt from this pipeline, using demonstrations with SS-CoT paths to guide the LLM in solving complex TQA tasks under In-Context Learning settings. Our experiments show that our approaches result in improved performance and reliability while being efficient. Our findings emphasize the importance of eliciting LLMs' reasoning capabilities to handle complex TQA tasks effectively.
Paper Structure (11 sections, 4 figures, 4 tables)

This paper contains 11 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Methods
  4. Tree structure
  5. The Seek-and-Solve pipeline
  6. The TQA-solving prompt
  7. Experiments
  8. Experimental Settings
  9. Results with the Seek-and-Solve pipeline
  10. Results with TQA-solving prompt
  11. Conclusion

Figures (4)

  • Figure 1: An illustration of different pipelines: (a) The pipeline used by existing works, characterized by simplifying the task; (b) The Seek-and-Solve pipeline, which incorporates Seek-CoT into the Solve stage in addition to generating the simplified task; (c) The Seek-and-Solve pipeline, which avoids generating the simplified task by incorporating Seek-CoT into the Solve stage to directly address the raw task; (d) The TQA-solving prompt, a compact single-step prompt.
  • Figure 2: Schematic of the Seek-and-Solve pipeline. The ICL prompts are exemplified with one-shot cases. Note, the table appears only in Solve-stage prompt.
  • Figure 3: Illustrations of complex table and tree structure.
  • Figure 4: Illustrations of TQA-solving prompt template and SS-CoT.