Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy

Chen Wang, Kaiyi Ji, Junyi Geng, Zhongqiang Ren, Taimeng Fu, Fan Yang, Yifan Guo, Haonan He, Xiangyu Chen, Zitong Zhan, Qiwei Du, Shaoshu Su, Bowen Li, Yuheng Qiu, Yi Du, Qihang Li, Yifan Yang, Xiao Lin, Zhipeng Zhao

TL;DR

<3-5 sentence high-level summary> Imperative Learning (IL) introduces a self-supervised, neuro-symbolic framework for robot autonomy by coupling a neural perception module, a symbolic reasoning engine, and a memory system within a specialized bilevel optimization (BLO) that enables reciprocal learning. The approach provides flexible optimization strategies (alternating, unrolled, implicit) to handle diverse lower-level solvers, including closed-form, first- and second-order, constrained, and discrete problems. Across five robot autonomy tasks—path planning, rule induction, optimal control, SLAM, and multi-robot routing—IL demonstrates superior generalization, efficiency, and robustness, often outperforming state-of-the-art baselines. The work also offers a practical blueprint for integrating neuro-symbolic reasoning with differentiable optimization and releases code to foster broader adoption.

Abstract

Data-driven methods such as reinforcement and imitation learning have achieved remarkable success in robot autonomy. However, their data-centric nature still hinders them from generalizing well to ever-changing environments. Moreover, labeling data for robotic tasks is often impractical and expensive. To overcome these challenges, we introduce a new self-supervised neuro-symbolic (NeSy) computational framework, imperative learning (IL), for robot autonomy, leveraging the generalization abilities of symbolic reasoning. The framework of IL consists of three primary components: a neural module, a reasoning engine, and a memory system. We formulate IL as a special bilevel optimization (BLO), which enables reciprocal learning over the three modules. This overcomes the label-intensive obstacles associated with data-driven approaches and takes advantage of symbolic reasoning concerning logical reasoning, physical principles, geometric analysis, etc. We discuss several optimization techniques for IL and verify their effectiveness in five distinct robot autonomy tasks including path planning, rule induction, optimal control, visual odometry, and multi-robot routing. Through various experiments, we show that IL can significantly enhance robot autonomy capabilities and we anticipate that it will catalyze further research across diverse domains.

Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy

TL;DR

<3-5 sentence high-level summary> Imperative Learning (IL) introduces a self-supervised, neuro-symbolic framework for robot autonomy by coupling a neural perception module, a symbolic reasoning engine, and a memory system within a specialized bilevel optimization (BLO) that enables reciprocal learning. The approach provides flexible optimization strategies (alternating, unrolled, implicit) to handle diverse lower-level solvers, including closed-form, first- and second-order, constrained, and discrete problems. Across five robot autonomy tasks—path planning, rule induction, optimal control, SLAM, and multi-robot routing—IL demonstrates superior generalization, efficiency, and robustness, often outperforming state-of-the-art baselines. The work also offers a practical blueprint for integrating neuro-symbolic reasoning with differentiable optimization and releases code to foster broader adoption.

Abstract

Data-driven methods such as reinforcement and imitation learning have achieved remarkable success in robot autonomy. However, their data-centric nature still hinders them from generalizing well to ever-changing environments. Moreover, labeling data for robotic tasks is often impractical and expensive. To overcome these challenges, we introduce a new self-supervised neuro-symbolic (NeSy) computational framework, imperative learning (IL), for robot autonomy, leveraging the generalization abilities of symbolic reasoning. The framework of IL consists of three primary components: a neural module, a reasoning engine, and a memory system. We formulate IL as a special bilevel optimization (BLO), which enables reciprocal learning over the three modules. This overcomes the label-intensive obstacles associated with data-driven approaches and takes advantage of symbolic reasoning concerning logical reasoning, physical principles, geometric analysis, etc. We discuss several optimization techniques for IL and verify their effectiveness in five distinct robot autonomy tasks including path planning, rule induction, optimal control, visual odometry, and multi-robot routing. Through various experiments, we show that IL can significantly enhance robot autonomy capabilities and we anticipate that it will catalyze further research across diverse domains.

Paper Structure

This paper contains 96 sections, 2 theorems, 51 equations, 18 figures, 15 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Bilevel Optimization
  4. Robot Learning Frameworks
  5. Imitation Learning
  6. Reinforcement learning
  7. Meta-learning
  8. Neuro-symbolic Learning
  9. Logical reasoning
  10. Physical reasoning
  11. Imperative Learning
  12. Structure
  13. Formulation
  14. Self-supervised Learning
  15. Memory
  16. ...and 81 more sections

Key Result

Lemma 1

Assume the LL cost $L(\cdot)$ and the constraint $\xi(\cdot)$ are 2-order differentiable near $({\bm{\theta}},{\bm{\gamma}},\bm \phi^*({\bm{\theta}},{\bm{\gamma}}))$ and the Hessian matrix $H$ below is invertible, we then have where the derivative matrices are given by and duel variable $\lambda\in\mathbb{R}$ satisfies $\lambda L_{\bm{\phi}} = \frac{\partial L\xi(f, g({\bm{\mu}}), M({\bm{\gamma}

Figures (18)

  • Figure 1: The framework of imperative learning (IL) consists of three primary modules including a neural perceptual network, a symbolic reasoning engine, and a general memory system. IL is formulated as a special BLO, enabling reciprocal learning and mutual correction among the three modules.
  • Figure 2: The framework of iA* search. The network predicts a confined search space, leading to overall improved efficiency. The A$^*$ search algorithm eliminates the label dependence, resulting in a self-supervised path planning framework.
  • Figure 3: The framework of iPlanner. The higher-level network predicts waypoints, which are interpolated by the lower-level optimization to ensure path continuity and smoothness.
  • Figure 4: The iLogic pipeline, which simultaneously conducts rule induction and high-dimensional data grounding.
  • Figure 5: One snapshot of the visual action prediction task of LogiCity benchmark. The next actions of each agent are reasoned by our iLogic based on groundings and learned rules.
  • ...and 13 more figures

Theorems & Definitions (2)

  • Lemma 1
  • Lemma 2