VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning

TL;DR

The paper tackles the challenge of planning in robotics by learning Neuro-Symbolic Predicates (NSPs) that provide task-specific, perceptually grounded abstractions. It proposes an online method that alternates predicate invention via VLMs, predicate scoring, and goal-directed exploration within a planner-driven loop to build an abstract world model characterized by predicates and high-level actions (HLAs). This approach is validated across five simulated robotic domains, showing superior sample efficiency, stronger generalization to unseen tasks, and improved interpretability compared with hierarchical reinforcement learning, VLM-based planning, and traditional symbolic predicates. The results demonstrate that autonomously learned NSPs can approach oracle-level performance with manually designed abstractions, enabling robust, long-horizon robot planning in visually rich and variable environments.

Abstract

Broadly intelligent agents should form task-specific abstractions that selectively expose the essential elements of a task, while abstracting away the complexity of the raw sensorimotor space. In this work, we present Neuro-Symbolic Predicates, a first-order abstraction language that combines the strengths of symbolic and neural knowledge representations. We outline an online algorithm for inventing such predicates and learning abstract world models. We compare our approach to hierarchical reinforcement learning, vision-language model planning, and symbolic predicate invention approaches, on both in- and out-of-distribution tasks across five simulated robotic domains. Results show that our approach offers better sample complexity, stronger out-of-distribution generalization, and improved interpretability.

Figures (7)

• Figure 1: Robot learning domains illustrating learned Neuro-Symbolic predicates. In (A) we learn a predicate that queries a VLM to check if a coffee jug is inside a coffee machine. In (B) we learn a predicate that checks if a balance beam is balanced. (Code lightly refactored to better fit in figure.)
• Figure 2: Example classifiers for Holding and OnPlateNSP.
• Figure 3: Environments. Top row: train task examples. Bottom row: evaluation task examples.
• Figure 4: Top row: percentage solved for different domains ($\uparrow$). Bottom: percentage of planning budget used to find the satisficing plans ($\downarrow$). The dashed line shows the minimal number of plans needed to solve all the tasks (1 plan per task).
• Figure 5: Example cropped current (right) and previous (left) observations used for interpreting ground predicates.