Can Stories Help LLMs Reason? Curating Information Space Through Narrative

TL;DR

Investigation of whether incorporating narrative elements can assist Large Language Models (LLMs) in solving complex problems more effectively finds that using various LLMs with SoT consistently surpasses using them with other techniques on physics, chemistry, math, and biology questions in both the GPQA and JEEBench datasets.

Abstract

Narratives are widely recognized as a powerful tool for structuring information and facilitating comprehension of complex ideas in various domains such as science communication. This paper investigates whether incorporating narrative elements can assist Large Language Models (LLMs) in solving complex problems more effectively. We propose a novel approach, Story of Thought (SoT), integrating narrative structures into prompting techniques for problem-solving. This approach involves constructing narratives around problem statements and creating a framework to identify and organize relevant information. Our experiments show that using various LLMs with SoT consistently surpasses using them with other techniques on physics, chemistry, math, and biology questions in both the GPQA and JEEBench datasets. The narrative-based information curation process in SoT enhances problem comprehension by contextualizing critical in-domain information and highlighting causal relationships within the problem space.

Figures (4)

• Figure 1: A high-level overview of Story of Thought (SoT), consisting of three steps (top): ① Question Clarification, ② Narrative Generation, ③ Solving Task and an actual example of LLM output (bottom) in each step for the GPQA task. The prompt designed for step 2 incorporates the narrative techniques (highlighted in blue) such as analogical reasoning, which identifies similarities between the target concept (information being conveyed) and a more familiar concept (analogy) and progressive disclosure which reveals information gradually throughout the narrative, rather than presenting it all at once. See \ref{['appsec:prompts']} for prompts for each step and \ref{['sot-cot-examples']} for an example.
• Figure 2: Performance of Story of Thought (SoT) on GPQA and JEEBench across various LLMs and domains.
• Figure 3: Correlation coefficients among all narrative techniques (PD = Progressive Disclosure, BR = Branching, AN = Analogy, AR = Analogical Reasoning, ME = Metaphor) used in the SoT approach for GPT 4 and Llama 3 70B in solved and unsolved tasks.
• Figure 4: An actual example of SoT.