Neuro-Symbolic Concepts

TL;DR

The paper tackles data efficiency, compositional generalization, continual learning, and cross-domain transfer in embodied AI by introducing a neuro-symbolic concept framework that grounds a typed vocabulary of objects, properties, relations, and actions in sensory and actuator data. It formalizes concepts as a mix of symbolic programs and neural embeddings, enabling modular composition and reusable reasoning across 2D/3D/robotic domains. The Neuro-Symbolic Concept Learner (NS-CL) and related methods demonstrate strong data-efficient grounding, robust compositional generalization, continual learning through curriculum-like and meta-learning approaches, and zero-shot transfer to new tasks and domains. This framework advances toward generalist, interpretable AI by enabling scalable cross-domain concept libraries and modular reasoning over perceptual and symbolic representations.

Abstract

This article presents a concept-centric paradigm for building agents that can learn continually and reason flexibly. The concept-centric agent utilizes a vocabulary of neuro-symbolic concepts. These concepts, such as object, relation, and action concepts, are grounded on sensory inputs and actuation outputs. They are also compositional, allowing for the creation of novel concepts through their structural combination. To facilitate learning and reasoning, the concepts are typed and represented using a combination of symbolic programs and neural network representations. Leveraging such neuro-symbolic concepts, the agent can efficiently learn and recombine them to solve various tasks across different domains, ranging from 2D images, videos, 3D scenes, and robotic manipulation tasks. This concept-centric framework offers several advantages, including data efficiency, compositional generalization, continual learning, and zero-shot transfer.

Figures (6)

• Figure 1: In a neuro-symbolic concept-centric framework, different concepts such as object categories, properties, relations, and actions are represented as a combination of programmatic and neural representations. In (a), the neural representation connects the concept with sensory and actuation representations. In (b), different concepts can be combined to form new compound concepts.
• Figure 2: Three challenges in (a): Compositional generalization. (b): Continual learning of concepts for reasoning. (c): Transfer learned concepts across domains.
• Figure 3: Humans learn visual concepts, words, and semantic parsing jointly and incrementally. I. Learning visual concepts (red vs. green) starts from looking at simple scenes, reading simple questions, and reasoning over contrastive examples fazly2010probabilistic. II. Afterwards, we can interpret referential expressions based on the learned object-based concepts, and learn relational concepts (e.g., on the right of, the same material as). III Finally, we can interpret complex questions from visual cues by exploiting the compositional structure.
• Figure 4: NS-CL uses neural symbolic reasoning to bridge the learning of visual concepts, words, and semantic parsing.
• Figure 5: The neuro-symbolic execution procedure of a program based on the visual representation and concept embeddings.