High-Degrees-of-Freedom Dynamic Neural Fields for Robot Self-Modeling and Motion Planning

TL;DR

This paper tackles robot self-modeling without depth information by learning a kinematic representation from pose-annotated 2D images. It proposes a high-DOFs dynamic neural density field with an encoder-based architecture and curricular sampling to model an 8-DOF configuration conditioned on joint states, using a single camera with base rotation for quasi-multi-view consistency. The method yields a neural-implicit full-body self-model that supports differentiable forward predictions and enables inverse kinematics and configuration-space planning, achieving a Chamfer-L2 of $1.94\%$ of the workspace in a $1.254$ m tall workspace on a $7$-DOF robot. This approach enables autonomous self-modeling and motion planning without depth data and opens avenues for dynamic object-centric scenes and multi-robot settings.

Abstract

A robot self-model is a task-agnostic representation of the robot's physical morphology that can be used for motion planning tasks in the absence of a classical geometric kinematic model. In particular, when the latter is hard to engineer or the robot's kinematics change unexpectedly, human-free self-modeling is a necessary feature of truly autonomous agents. In this work, we leverage neural fields to allow a robot to self-model its kinematics as a neural-implicit query model learned only from 2D images annotated with camera poses and configurations. This enables significantly greater applicability than existing approaches which have been dependent on depth images or geometry knowledge. To this end, alongside a curricular data sampling strategy, we propose a new encoder-based neural density field architecture for dynamic object-centric scenes conditioned on high numbers of degrees of freedom (DOFs). In a 7-DOF robot test setup, the learned self-model achieves a Chamfer-L2 distance of 2% of the robot's workspace dimension. We demonstrate the capabilities of this model on motion planning tasks as an exemplary downstream application.

Figures (4)

• Figure 1: Overview of contributions (shaded): When a kinematic model is unavailable for the robot, 1) our method to collect curricular annotated depth-free image data can be used instead to train 2) a high-DOFs dynamic neural density field which the robot uses as a self-model. 3) Its forward and inverse kinematics capabilities enable motion planning applications.
• Figure 2: Overview of proposed method: A) Training images of robot in different configurations: While the point coordinates and annotated configuration are the inputs to the neural network, the true color of the pixel is used for the reconstruction loss. B) Neural network architecture: The DOF values and spatial coordinates are individually encoded, concatenated and group-wise encoded, and concatenated and processed to an output density. C) The trained neural density field is used to evaluate the validity of a configuration relative to an obstacle by predicting the densities of points queried from its volume.
• Figure 3: Self-model results: Predicted vs. ground-truth meshes, smoothed and reconstructed via marching cubes from point clouds generated by querying the high-DOFs neural density field in the given random test configuration. The configurations are shown in radians. Please also see suppl. video video.
• Figure 4: Motion planning results: A) Joint value optimization via input PGD. A density loss is minimized when the sphere (green) is touched. B) RRT planning in config. space to intersect a point (green). The query model rejects samples with density on the obstacle (red). Please also see suppl. video video.