WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents

TL;DR

WALL-E 2.0 tackles the gap between LLM priors and environment dynamics by introducing a training-free neuro-symbolic learning pipeline that extracts executable code rules from exploration trajectories. By integrating action rules, knowledge graphs, and scene graphs into a neurosymbolic world model and feeding it into a model-predictive control loop, the approach achieves verifiable, efficient planning for LLM-based agents in open-world tasks. Empirical results on Mars and ALFWorld show substantial improvements over RL-based and prior LLM-based methods, including notable gains in reward, score, and success rate, and a new record in ALFWorld after only a few iterations. The work demonstrates the viability and practicality of world-aligned LLM agents, with strong implications for rapid adaptation and safe, reliable planning in complex environments, while outlining avenues for handling stochastic dynamics in future work.

Abstract

Can we build accurate world models out of large language models (LLMs)? How can world models benefit LLM agents? The gap between the prior knowledge of LLMs and the specified environment's dynamics usually bottlenecks LLMs' performance as world models. To bridge the gap, we propose a training-free "world alignment" that learns an environment's symbolic knowledge complementary to LLMs. The symbolic knowledge covers action rules, knowledge graphs, and scene graphs, which are extracted by LLMs from exploration trajectories and encoded into executable codes to regulate LLM agents' policies. We further propose an RL-free, model-based agent "WALL-E 2.0" through the model-predictive control (MPC) framework. Unlike classical MPC requiring costly optimization on the fly, we adopt an LLM agent as an efficient look-ahead optimizer of future steps' actions by interacting with the neurosymbolic world model. While the LLM agent's strong heuristics make it an efficient planner in MPC, the quality of its planned actions is also secured by the accurate predictions of the aligned world model. They together considerably improve learning efficiency in a new environment. On open-world challenges in Mars (Minecraft like) and ALFWorld (embodied indoor environments), WALL-E 2.0 significantly outperforms existing methods, e.g., surpassing baselines in Mars by 16.1%-51.6% of success rate and by at least 61.7% in score. In ALFWorld, it achieves a new record 98% success rate after only 4 iterations.

Figures (6)

• Figure 1: WALL-E 2.0 mining a diamond on Mars. Step 1-2: The agent makes decisions via MPC with the initial unaligned world model, resulting in a failed action for mining iron. Step 3: leveraging previous trajectories and world model predictions, WALL-E 2.0 learns symbolic knowledge, including rules, knowledge graphs, and scene graphs. Step 4-5: The learned symbolic knowledge helps the world model make accurate predictions and correct the previous mistake. Step 6-7: The agent adjusts its decision accordingly and replaces stone pickaxe with iron pickaxe toward completing the task.
• Figure 2: Overview of WALL-E 2.0. The agent determines actions to take via MPC, which optimizes future steps' actions by interacting with a neurosymbolic world model. The world model adopts an LLM whose predictions are aligned with environment dynamics through code rules converted from symbolic knowledge (action rules, knowledge/scene graph) learned via inductive reasoning from real trajectories and predicted trajectories.
• Figure 3: NeuroSymbolic Learning of Code Rules. WALL-E 2.0 iteratively refines the symbolic knowledge with the agent's actual trajectories in the environment and the world model's predicted trajectories. The NeuroSymbolic learning takes 4 stages: (1) comparing predicted and actual trajectories; (2) learning new symbolic knowledge from real trajectories; (4) translating symbolic knowledge to code; and (4) Code rule set pruning via solving a maximum coverage problem.
• Figure 4: WALL-E 2.0 vs. baselines on 134 testing tasks from the ALFWorld. WALL-E 2.0 exhibiting superior planning ability and achieves the highest success rate after only 4 iterations, significantly surpassing other baselines.
• Figure 5: Comparison between WALL-E 2.0 and IfR (the best baseline in Table \ref{['tab:mars-main-result']}) over learning iterations in Mars. WALL-E 2.0 achieves a clear advantage over IfR in both learning efficiency and overall performance, due to the world alignment with diverse symbolic knowledge and code rules.