Solving Situation Puzzles with Large Language Model and External Reformulation

TL;DR

This work addresses the difficulty of large language models in multi-turn, interactive reasoning required by situation puzzles. It introduces an external reformulation approach that aggregates prior Q&A interactions to reformulate the puzzle description and restarts the chat, thereby providing a refreshed starting point for reasoning. A priority-based selection of Q&A pairs informs the reformulation, and the method is evaluated on five puzzles against a baseline single-chat approach. Empirical results show improved win rates and more efficient questioning, suggesting that strategic problem reformulation can enhance interactive reasoning capabilities in complex tasks with LLMs.

Abstract

In recent years, large language models (LLMs) have shown an impressive ability to perform arithmetic and symbolic reasoning tasks. However, we found that LLMs (e.g., ChatGPT) cannot perform well on reasoning that requires multiple rounds of dialogue, especially when solving situation puzzles. Specifically, LLMs intend to ask very detailed questions focusing on a specific aspect or same/similar questions after several rounds of Q&As. To help LLMs get out of the above dilemma, we propose a novel external reformulation methodology, where the situation puzzle will be reformulated after several rounds of Q&A or when the LLMs raise an incorrect guess. Experiments show superior performance (e.g., win rate, number of question/guess attempts) of our method than directly using LLMs for solving situation puzzles, highlighting the potential of strategic problem reformulation to enhance the reasoning capabilities of LLMs in complex interactive scenarios.

Figures (5)

• Figure 1: An example of solving a situation puzzle, including several rounds of interaction between the host and the player.
• Figure 2: Two examples of directly using LLMs to solve situation puzzles. For the example in the top, after several rounds of Q&As, the player intends to ask very detailed questions focusing on a specific aspect. For the example at the bottom, the player asks the same or similar questions after several rounds of Q&As.
• Figure 3: The key idea of our proposed reformulation manner for LLMs to solve the situation puzzles.
• Figure 4: We conducted experiments on the above five situation puzzles.
• Figure 5: Case study. In the first chat session, the game starts with the host giving a description of the situation. After 5 rounds of Q&As, two Yes-questions and the first No-question are selected to generate the hints. To reformulate, hints are integrated into the description prompt and a new chat session starts with the new description. In this case, the game ends in the second chat session as the player gives a correct guess and finally wins the game.