Evaluating World Models with LLM for Decision Making

TL;DR

This work tackles the problem of evaluating world models used for decision making when powered by large language models. It proposes a decision-focused evaluation framework with three tasks—policy verification, action proposal, and policy planning—and validates GPT-4o and GPT-4o-mini across 31 diverse, rule-based environments. The study reveals that GPT-4o generally outperforms the smaller model, but long-horizon decision making and cross-module coupling introduce notable instability. The results offer practical guidance for designing and evaluating LLM-based world models and highlight avenues to enhance domain knowledge integration and planning reliability.

Abstract

World model emerges as a key module in decision making, where MuZero and Dreamer achieve remarkable successes in complex tasks. Recent work leverages Large Language Models (LLMs) as general world simulators to simulate the dynamics of the world due to their generalizability. LLMs also serve as the world model for deliberative reasoning in Reasoning via Planning (RAP) and Tree of Thought (ToT). However, the world models are either evaluated as a general world simulator, or as a functional module of the agent, i.e., predicting the transitions to assist the planning. In this work, we propose a comprehensive evaluation of the world models with LLMs from the decision making perspective. Specifically, we leverage the 31 diverse environments from (Wang et al., 2023;2024) and curate the rule-based policy of each environment for the diverse evaluation. Then, we design three main tasks, i.e., policy verification, action proposal, and policy planning, where the world models can be used for decision making solely. Finally, we conduct the comprehensive evaluation of the advanced LLMs, i.e., GPT-4o and GPT-4o-mini, on the environments for the three main tasks under various settings. The key observations include: i) GPT-4o significantly outperforms GPT-4o-mini on the three main tasks, especially for the tasks which require the domain knowledge, ii) the performance of the world model with LLM will be decreased for long-term decision-making tasks, and iii) the combination of different functionalities of the world model will brings additional unstabilities of the performance.

Figures (13)

• Figure 1: Picking the action with higher value. The prediction of B in Case 2 is more accurate than Case 1 in term of the $L_2$ loss, but leads to the wrong action.
• Figure 2: Evaluation of World Model with LLM for Decision Making.
• Figure 3: The next state prediction and reward/terminal prediction are considered in wang2024language. With the two predictions, we can complete policy verification. We introduce the action proposal of the world model and the policy planning can be completed with the policy verification and action proposal.
• Figure 4: Number of steps to complete the tasks
• Figure 5: The accuracy of the world model to verify the correct policies