Logic-of-Thought: Empowering Large Language Models with Logic Programs for Solving Puzzles in Natural Language
Naiqi Li, Peiyuan Liu, Zheng Liu, Tao Dai, Yong Jiang, Shu-Tao Xia
TL;DR
Logot introduces a hybrid framework that grounds large language models in declarative reasoning by translating natural-language puzzle rules and states into Answer Set Programs. An ASP interpreter then performs exact, exhaustive search to yield solutions, enabling near-perfect accuracy on both grid puzzles and dynamic planning puzzles. The approach leverages two in-context learning modules to translate rules and initial states, followed by an ASP-based inference and a lightweight postprocessing step to present human-readable answers. With a public benchmark and data, Logot demonstrates a scalable, accurate path for solving complex puzzles and potentially other reasoning tasks requiring precise search and knowledge grounding.
Abstract
Solving puzzles in natural language poses a long-standing challenge in AI. While large language models (LLMs) have recently shown impressive capabilities in a variety of tasks, they continue to struggle with complex puzzles that demand precise reasoning and exhaustive search. In this paper, we propose Logic-of-Thought (Logot), a novel framework that bridges LLMs with logic programming to address this problem. Our method leverages LLMs to translate puzzle rules and states into answer set programs (ASPs), the solution of which are then accurately and efficiently inferred by an ASP interpreter. This hybrid approach combines the natural language understanding of LLMs with the precise reasoning capabilities of logic programs. We evaluate our method on various grid puzzles and dynamic puzzles involving actions, demonstrating near-perfect accuracy across all tasks. Our code and data are available at: https://github.com/naiqili/Logic-of-Thought.