NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domains

TL;DR

NeSyC tackles the challenge of generalizing actionable knowledge for embodied agents in open domains by coupling Large Language Models with symbolic tools within a hypothetico-deductive framework. It introduces two core mechanisms: a contrastive generality improvement scheme that generates hypotheses with LLMs and validates them via symbolic solvers, and a memory-based monitoring scheme that triggers knowledge refinement upon action errors. The approach iteratively reformulates generalized rules and then applies them through an ASP-based planner, guided by ILP and ASP feedback, enabling continual knowledge evolution. Empirical results across ALFWorld, VirtualHome, Minecraft, RLBench, and real-world robotics show significant performance gains and robustness to dynamic environments, illustrating practical potential for open-domain embodied AI. The work also analyzes robustness to incomplete experiences, ablation effects, and feedback modalities, outlining avenues for efficiency improvements and safety considerations in real deployments.

Abstract

We explore neuro-symbolic approaches to generalize actionable knowledge, enabling embodied agents to tackle complex tasks more effectively in open-domain environments. A key challenge for embodied agents is the generalization of knowledge across diverse environments and situations, as limited experiences often confine them to their prior knowledge. To address this issue, we introduce a novel framework, NeSyC, a neuro-symbolic continual learner that emulates the hypothetico-deductive model by continually formulating and validating knowledge from limited experiences through the combined use of Large Language Models (LLMs) and symbolic tools. Specifically, we devise a contrastive generality improvement scheme within NeSyC, which iteratively generates hypotheses using LLMs and conducts contrastive validation via symbolic tools. This scheme reinforces the justification for admissible actions while minimizing the inference of inadmissible ones. Additionally, we incorporate a memory-based monitoring scheme that efficiently detects action errors and triggers the knowledge refinement process across domains. Experiments conducted on diverse embodied task benchmarks-including ALFWorld, VirtualHome, Minecraft, RLBench, and a real-world robotic scenario-demonstrate that NeSyC is highly effective in solving complex embodied tasks across a range of open-domain environments.

Figures (15)

• Figure 1: The concept of $\textsc{NeSyC}\xspace$. In the leftmost part, domain shift leads the agent to fail by trying to grasp an oversized drawer’s broad surface, which is infeasible. The remaining parts contrast two approaches: Case 1 treats LLMs and symbolic tools as separate functions for semantic parsing and logical reasoning, while Case 2 integrates them into a collaborative process, enabling $\textsc{NeSyC}\xspace$ to generalize actionable knowledge and compute logically valid actions for open-domain environment.
• Figure 2: The structure of $\textsc{NeSyC}\xspace$. $\textsc{NeSyC}\xspace$ iterates (i) Rule Reformulation and (ii) Rule Application phases. In (i), generalized knowledge $R$ is reformulated via contrastive generality improvement. In (ii), $R$ is applied and continually adapted to the environment via memory-based monitoring.
• Figure 3: Contrastive generality improvement scheme evaluation on environments. SR, GC, and HI measure the performance of the generalized knowledge $R$ on the first interpretation step. SR, GC and HI report scores after iterative adjustment.
• Figure 4: F1 score evaluation on predicate categories. Colors indicating LLMs are consistent with those used for different LLMs in Table \ref{['tab:ana:llm']}
• Figure 5: Real-world desk rearrangement tasks. Initially, $\textsc{NeSyC}\xspace$ does not include knowledge for picking up Hanoi blocks from experiences. After failures, $\textsc{NeSyC}\xspace$ refines to enhance grasping capabilities, enabling the robot to successfully complete the desk rearrangement task.