Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Need Consultants for Reasoning: Becoming an Expert in a Complex Human System Through Behavior Simulation

Chuwen Wang, Shirong Zeng, Cheng Wang

TL;DR

A novel reasoning framework, termed ``Mosaic Expert Observation Wall'' (MEOW) exploiting generative-agents-based simulation technique, is proposed that can cooperate with existing methodologies to enhance the reasoning abilities of LLMs in complex human systems.

Abstract

Large language models (LLMs), in conjunction with various reasoning reinforcement methodologies, have demonstrated remarkable capabilities comparable to humans in fields such as mathematics, law, coding, common sense, and world knowledge. In this paper, we delve into the reasoning abilities of LLMs within complex human systems. We propose a novel reasoning framework, termed ``Mosaic Expert Observation Wall'' (MEOW) exploiting generative-agents-based simulation technique. In the MEOW framework, simulated data are utilized to train an expert model concentrating ``experience'' about a specific task in each independent time of simulation. It is the accumulated ``experience'' through the simulation that makes for an expert on a task in a complex human system. We conduct the experiments within a communication game that mirrors real-world security scenarios. The results indicate that our proposed methodology can cooperate with existing methodologies to enhance the reasoning abilities of LLMs in complex human systems.

Large Language Models Need Consultants for Reasoning: Becoming an Expert in a Complex Human System Through Behavior Simulation

TL;DR

A novel reasoning framework, termed ``Mosaic Expert Observation Wall'' (MEOW) exploiting generative-agents-based simulation technique, is proposed that can cooperate with existing methodologies to enhance the reasoning abilities of LLMs in complex human systems.

Abstract

Large language models (LLMs), in conjunction with various reasoning reinforcement methodologies, have demonstrated remarkable capabilities comparable to humans in fields such as mathematics, law, coding, common sense, and world knowledge. In this paper, we delve into the reasoning abilities of LLMs within complex human systems. We propose a novel reasoning framework, termed ``Mosaic Expert Observation Wall'' (MEOW) exploiting generative-agents-based simulation technique. In the MEOW framework, simulated data are utilized to train an expert model concentrating ``experience'' about a specific task in each independent time of simulation. It is the accumulated ``experience'' through the simulation that makes for an expert on a task in a complex human system. We conduct the experiments within a communication game that mirrors real-world security scenarios. The results indicate that our proposed methodology can cooperate with existing methodologies to enhance the reasoning abilities of LLMs in complex human systems.
Paper Structure (26 sections, 6 equations, 5 figures, 3 tables)

This paper contains 26 sections, 6 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Reasoning
  4. Generative-agents-based Simulation
  5. Behavior Simulation in Communication Game
  6. Game Rules
  7. The Scope of Behavior Simulation
  8. Instance in "Find The Spy"
  9. Architecture and Methodology
  10. Game Simulator
  11. LLM Agent
  12. Game Engine
  13. Implementation of MEOW
  14. Simulated Data to Heterogeneous Graph
  15. Expert Model Design
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Mosaic investigation wall is what detectives use to visualize clues and interactions within cases. Through simulating current individuals' behaviors and interactions on a Mosaic wall, a detective can deduce who may be the crime in corresponding real case.
  • Figure 2: The illustration of implementing generative-agents-based simulation and training an expert model based on simulated data in the four-player version of "Find The Spy".
  • Figure 3: Example of heterogeneous graph. The numerical labels $i$ at the tail of each arrow represent the $i$th round of the game.
  • Figure 4: The architecture of expert models.
  • Figure 5: The framework of MEOW in the four-player version of "Find The Spy".