Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models

Daniel Fleischer, Moshe Berchansky, Gad Markovits, Moshe Wasserblat

TL;DR

SQuARE addresses limitations of traditional chain-of-thought prompting by introducing a self-interrogation loop that prompts LLMs to generate and answer multiple sub-questions before the main query. The method integrates with existing prompting paradigms and is controlled by a tunable parameter $N$ for the number of sub-questions. Evaluations on TriviaQA, HotpotQA, and ASQA using Llama-3 models and GPT-4o show that SQuARE outperforms standard CoT and RaR baselines, with pronounced gains on smaller models. The work provides ablation analyses of $N$, few-shot examples, and aggregation strategies, and releases code at the provided GitHub link, highlighting practical potential for improved reasoning in real-world QA systems.

Abstract

In the rapidly evolving field of Natural Language Processing, Large Language Models (LLMs) are tasked with increasingly complex reasoning challenges. Traditional methods like chain-of-thought prompting have shown promise but often fall short in fully leveraging a model's reasoning capabilities. This paper introduces SQuARE (Sequential Question Answering Reasoning Engine), a novel prompting technique designed to improve reasoning through a self-interrogation paradigm. Building upon CoT frameworks, SQuARE prompts models to generate and resolve multiple auxiliary questions before tackling the main query, promoting a more thorough exploration of various aspects of a topic. Our expansive evaluations, conducted with Llama 3 and GPT-4o models across multiple question-answering datasets, demonstrate that SQuARE significantly surpasses traditional CoT prompts and existing rephrase-and-respond methods. By systematically decomposing queries, SQuARE advances LLM capabilities in reasoning tasks. The code is publicly available at https://github.com/IntelLabs/RAG-FiT/tree/square.

SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models

TL;DR

SQuARE addresses limitations of traditional chain-of-thought prompting by introducing a self-interrogation loop that prompts LLMs to generate and answer multiple sub-questions before the main query. The method integrates with existing prompting paradigms and is controlled by a tunable parameter for the number of sub-questions. Evaluations on TriviaQA, HotpotQA, and ASQA using Llama-3 models and GPT-4o show that SQuARE outperforms standard CoT and RaR baselines, with pronounced gains on smaller models. The work provides ablation analyses of , few-shot examples, and aggregation strategies, and releases code at the provided GitHub link, highlighting practical potential for improved reasoning in real-world QA systems.

Abstract

In the rapidly evolving field of Natural Language Processing, Large Language Models (LLMs) are tasked with increasingly complex reasoning challenges. Traditional methods like chain-of-thought prompting have shown promise but often fall short in fully leveraging a model's reasoning capabilities. This paper introduces SQuARE (Sequential Question Answering Reasoning Engine), a novel prompting technique designed to improve reasoning through a self-interrogation paradigm. Building upon CoT frameworks, SQuARE prompts models to generate and resolve multiple auxiliary questions before tackling the main query, promoting a more thorough exploration of various aspects of a topic. Our expansive evaluations, conducted with Llama 3 and GPT-4o models across multiple question-answering datasets, demonstrate that SQuARE significantly surpasses traditional CoT prompts and existing rephrase-and-respond methods. By systematically decomposing queries, SQuARE advances LLM capabilities in reasoning tasks. The code is publicly available at https://github.com/IntelLabs/RAG-FiT/tree/square.

Paper Structure

This paper contains 16 sections, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. SQuARE
  3. Experiments
  4. Datasets
  5. Models
  6. Configurations
  7. Results
  8. Ablation Study
  9. Number of Generated Questions:
  10. Impact of Few-Shot Examples:
  11. Aggregation Methods:
  12. Related Works
  13. Conclusions and Summary
  14. Implementation Details
  15. Datasets
  16. ...and 1 more sections

Figures (2)

  • Figure 1: The SQuARE methods prompts the model to generate $N$ question-answer pairs about the topic and then respond to the original query, having established additional context.
  • Figure 2: Ablation study illustrating how few-shot examples influence performance metrics for the CoT, RaR, and SQuARE approaches, using the Llama-3.1 8B model.