Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment

Patricia Delafuente, Arya Honraopatil, Lara J. Martin

TL;DR

The paper evaluates whether reasoning helps LLM agents generate Avrae commands for DnD combat by comparing an instruction-tuned LLaMA-3.1-8B-Instruct model to a reasoning DeepSeek-R1-Distill-LLaMA-8B across five prompts. It introduces Format Check and Reference Check metrics to assess syntactic correctness and correct entity references, revealing that instruction-following models, with careful prompt design, can match or exceed reasoning models for this task, and that explicit CoT can hurt smaller models. The findings offer practical guidance: favor zero-shot or instruction-driven prompts with explicit formatting rules, and avoid forcing chain-of-thought reasoning for compact, action-generation tasks. This work advances grounded, structured language grounding for interactive DnD automation and informs prompt design choices for similar structured-command generation problems.

Abstract

This paper explores the application of Large Language Models (LLMs) and reasoning to predict Dungeons & Dragons (DnD) player actions and format them as Avrae Discord bot commands. Using the FIREBALL dataset, we evaluated a reasoning model, DeepSeek-R1-Distill-LLaMA-8B, and an instruct model, LLaMA-3.1-8B-Instruct, for command generation. Our findings highlight the importance of providing specific instructions to models, that even single sentence changes in prompts can greatly affect the output of models, and that instruct models are sufficient for this task compared to reasoning models.

Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment

TL;DR

The paper evaluates whether reasoning helps LLM agents generate Avrae commands for DnD combat by comparing an instruction-tuned LLaMA-3.1-8B-Instruct model to a reasoning DeepSeek-R1-Distill-LLaMA-8B across five prompts. It introduces Format Check and Reference Check metrics to assess syntactic correctness and correct entity references, revealing that instruction-following models, with careful prompt design, can match or exceed reasoning models for this task, and that explicit CoT can hurt smaller models. The findings offer practical guidance: favor zero-shot or instruction-driven prompts with explicit formatting rules, and avoid forcing chain-of-thought reasoning for compact, action-generation tasks. This work advances grounded, structured language grounding for interactive DnD automation and informs prompt design choices for similar structured-command generation problems.

Abstract

This paper explores the application of Large Language Models (LLMs) and reasoning to predict Dungeons & Dragons (DnD) player actions and format them as Avrae Discord bot commands. Using the FIREBALL dataset, we evaluated a reasoning model, DeepSeek-R1-Distill-LLaMA-8B, and an instruct model, LLaMA-3.1-8B-Instruct, for command generation. Our findings highlight the importance of providing specific instructions to models, that even single sentence changes in prompts can greatly affect the output of models, and that instruct models are sufficient for this task compared to reasoning models.
Paper Structure (21 sections, 1 figure, 1 table)

This paper contains 21 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Models
  4. LLMs for DnD
  5. Experimental Setup
  6. Data
  7. Models
  8. Evaluation Metrics
  9. Format Check.
  10. Reference Check.
  11. Prompting Experiments
  12. Prompt 1: Attack Example
  13. Prompt 2: Attack and Cast Example, Do not refer to examples
  14. Prompt 3: Attack and Cast Example, Do not refer to examples, No Explicit Reasoning
  15. Prompt 4: Zero-shot, Explicit Reasoning
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Reference Check percentage across all prompts for both models. Higher is better.