Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Are Neurosymbolic Reasoners

Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, Jun Wang

TL;DR

This work investigates using large language models as neurosymbolic reasoners for symbolic tasks in text-based games, by coupling LLMs with external symbolic modules and prompting strategies in a zero-shot setting. A role-initialization and action-query framework grounds the LLM's decisions while modules such as arithmetic, map navigation, and knowledge bases provide precise symbolic support. Empirical results on Arithmetic, MapReader, Sorting, and Text World Common Sense show that the LLM agent with symbolic modules achieves strong average performance (up to 88% across tasks) and often outperforms baselines like DRRN and Behavior Cloned Transformer without requiring extensive expert data; GPT-4 further improves performance on some tasks. The findings demonstrate the viability of LLMs as neurosymbolic reasoners for symbolic reasoning in interactive contexts and point to future work on richer prompts and more capable symbolic modules to broaden applicability.

Abstract

A wide range of real-world applications is characterized by their symbolic nature, necessitating a strong capability for symbolic reasoning. This paper investigates the potential application of Large Language Models (LLMs) as symbolic reasoners. We focus on text-based games, significant benchmarks for agents with natural language capabilities, particularly in symbolic tasks like math, map reading, sorting, and applying common sense in text-based worlds. To facilitate these agents, we propose an LLM agent designed to tackle symbolic challenges and achieve in-game objectives. We begin by initializing the LLM agent and informing it of its role. The agent then receives observations and a set of valid actions from the text-based games, along with a specific symbolic module. With these inputs, the LLM agent chooses an action and interacts with the game environments. Our experimental results demonstrate that our method significantly enhances the capability of LLMs as automated agents for symbolic reasoning, and our LLM agent is effective in text-based games involving symbolic tasks, achieving an average performance of 88% across all tasks.

Large Language Models Are Neurosymbolic Reasoners

TL;DR

This work investigates using large language models as neurosymbolic reasoners for symbolic tasks in text-based games, by coupling LLMs with external symbolic modules and prompting strategies in a zero-shot setting. A role-initialization and action-query framework grounds the LLM's decisions while modules such as arithmetic, map navigation, and knowledge bases provide precise symbolic support. Empirical results on Arithmetic, MapReader, Sorting, and Text World Common Sense show that the LLM agent with symbolic modules achieves strong average performance (up to 88% across tasks) and often outperforms baselines like DRRN and Behavior Cloned Transformer without requiring extensive expert data; GPT-4 further improves performance on some tasks. The findings demonstrate the viability of LLMs as neurosymbolic reasoners for symbolic reasoning in interactive contexts and point to future work on richer prompts and more capable symbolic modules to broaden applicability.

Abstract

A wide range of real-world applications is characterized by their symbolic nature, necessitating a strong capability for symbolic reasoning. This paper investigates the potential application of Large Language Models (LLMs) as symbolic reasoners. We focus on text-based games, significant benchmarks for agents with natural language capabilities, particularly in symbolic tasks like math, map reading, sorting, and applying common sense in text-based worlds. To facilitate these agents, we propose an LLM agent designed to tackle symbolic challenges and achieve in-game objectives. We begin by initializing the LLM agent and informing it of its role. The agent then receives observations and a set of valid actions from the text-based games, along with a specific symbolic module. With these inputs, the LLM agent chooses an action and interacts with the game environments. Our experimental results demonstrate that our method significantly enhances the capability of LLMs as automated agents for symbolic reasoning, and our LLM agent is effective in text-based games involving symbolic tasks, achieving an average performance of 88% across all tasks.
Paper Structure (26 sections, 2 figures, 7 tables)

This paper contains 26 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Decision Making.
  4. Text-based Game.
  5. Neurosymbolic Reasoning.
  6. Preliminaries
  7. Text-based Games as POMDPs.
  8. Symbolic Tasks.
  9. Methodology
  10. Playing Games with Symbolic Tasks
  11. Symbolic Modules.
  12. LLM as the Neurosymbolic Reasoner
  13. Role Initialization.
  14. Action Query.
  15. Experiments
  16. ...and 11 more sections

Figures (2)

  • Figure 1: The LLM agent is capable of interacting with the game environment, leveraging its reasoning abilities to determine the most suitable actions. These actions alter the environment's state and contribute to achieving the given objective. The environment, along with its corresponding symbolic modules, offers a valid set of actions to the LLM agent. The agent's responsibility is to select an action from this set. The chosen action will then dictate how the agent interacts with either the game environment or the symbolic module.
  • Figure 2: An overview of how an LLM agent plays text-based games with external symbolic modules. The following procedural steps are involved in utilizing the LLM agent for engaging in a text-based game. Initially, the LLM agent is provided with a role initialization prompt. The first observation received by the LLM agent comes from the text game environment. As depicted in the diagram, the selection of actions, determined by the LLM's reasoning, activates the symbolic module. Subsequently, the symbolic module provides output, including observations related to the module. Then the next action chosen by the LLM agent is influenced by the outcome from the symbolic module. This process is executed repeatedly until the end of the game.