Digital Player: Evaluating Large Language Models based Human-like Agent in Games

TL;DR

The paper presents CivSim, a testbed built on Unciv to evaluate LLM-based human-like agents as digital players. It introduces CivAgent, a memory- and RAG-enabled architecture augmented with a game simulator and two reflection mechanisms to support long-horizon reasoning and social interaction, facilitated by a Discord interface. Through multi-variant experiments across several LLMs and CivAgent configurations, the study shows that incorporating memory, reflection, and simulation improves strategic planning and diplomacy, while also highlighting strengths in negotiation and deception tasks for state-of-the-art models. The framework, inspired by citizen science, offers a low-cost data flywheel for collecting human feedback and scaling agent capabilities, with potential for broader applicability across domains. The work provides open-source resources for researchers to build and extend LLM-based human-like agents in interactive, multi-turn environments.

Abstract

With the rapid advancement of Large Language Models (LLMs), LLM-based autonomous agents have shown the potential to function as digital employees, such as digital analysts, teachers, and programmers. In this paper, we develop an application-level testbed based on the open-source strategy game "Unciv", which has millions of active players, to enable researchers to build a "data flywheel" for studying human-like agents in the "digital players" task. This "Civilization"-like game features expansive decision-making spaces along with rich linguistic interactions such as diplomatic negotiations and acts of deception, posing significant challenges for LLM-based agents in terms of numerical reasoning and long-term planning. Another challenge for "digital players" is to generate human-like responses for social interaction, collaboration, and negotiation with human players. The open-source project can be found at https:/github.com/fuxiAIlab/CivAgent.

Figures (6)

• Figure 1: Examples of interfaces of Unciv game and Discord chatbots.
• Figure 2: The architecture of the LLM-based CivAgent. It involves a decision-making process that can be delineated into the following steps: 1. Retrieve the current turn's game save file and extract observations; 2. Integrate game context from observations, dialogue history, historical events, etc., and employ RAG to retrieve experience required for prompts; 3-6. Based on observations and information provided by memory, CivAgent engages in reasoning, with permission to utilize tools like game simulators if necessary. Using this reasoned information, CivAgent formulates long-term and short-term plans within the skills spaces. After several turns or game end, CivAgent reflects on its game situation and performance, and stores reflective experiences into long-term memory; 7. CivAgent then translates short-term planning (i.e., sequences of skills with filled parameters) into action interfaces for execution within the game environment.
• Figure 3: A simplified diagram of CivAgent from the perspective of the prompt.
• Figure 4: (a) A schematic diagram illustrating the procedural steps for CivAgent's actions in each round, where robots of different colors represent different civilizations. The steps are 1-2. Using LLM to determine which skills to activate; 3-4. The game server calls the agent to use skills; 5. If there is a decision request, it is sent to the corresponding agent; 6-7. Use LLM to make decisions about requests. (b) The engineering architecture diagram of CivSim. The game client synchronizes data with the AI module through a save file. The AI module makes decisions on different aspects in a specific order. The CivAgent and multiplayer game server, coded in Python and easily modifiable, are shown in the bottom right corner of the diagram.
• Figure 5: Diagram of decision logic involved in the simulator.