Growable and Interpretable Neural Control with Online Continual Learning for Autonomous Lifelong Locomotion Learning Machines

TL;DR

This work presents GOLLUM, a growable online locomotion learning framework for autonomous lifelong learning that jointly addresses incomprehensibility, sample inefficiency, knowledge exploitation, and catastrophic forgetting. The method combines an interpretable neural control with two interpretation dimensions (column-wise skill columns and layer-wise modular layers), a dual layer learning mechanism (primary skill refinement and supplementary cross-skill exploitation), and subnetwork neurogenesis to incrementally acquire new skills. On a MORF hexapod, GOLLUM demonstrates rapid primitive locomotion on flat terrain, online continual learning across slopes, deformable terrains, and motor faults, and the ability to recall and recombine prior skills with minimal forgetting, all under simple reward signals. The approach outperforms state-of-the-art baselines in learning speed, enables explicit exploitation of similarity across tasks, and provides quantitative interpretability metrics, suggesting practical pathways for real-world, autonomous lifelong robotic learning and broader applicability beyond locomotion.

Abstract

Continual locomotion learning faces four challenges: incomprehensibility, sample inefficiency, lack of knowledge exploitation, and catastrophic forgetting. Thus, this work introduces Growable Online Locomotion Learning Under Multicondition (GOLLUM), which exploits the interpretability feature to address the aforementioned challenges. GOLLUM has two dimensions of interpretability: layer-wise interpretability for neural control function encoding and column-wise interpretability for robot skill encoding. With this interpretable control structure, GOLLUM utilizes neurogenesis to unsupervisely increment columns (ring-like networks); each column is trained separately to encode and maintain a specific primary robot skill. GOLLUM also transfers the parameters to new skills and supplements the learned combination of acquired skills through another neural mapping layer added (layer-wise) with online supplementary learning. On a physical hexapod robot, GOLLUM successfully acquired multiple locomotion skills (e.g., walking, slope climbing, and bouncing) autonomously and continuously within an hour using a simple reward function. Furthermore, it demonstrated the capability of combining previous learned skills to facilitate the learning process of new skills while preventing catastrophic forgetting. Compared to state-of-the-art locomotion learning approaches, GOLLUM is the only approach that addresses the four challenges above mentioned without human intervention. It also emphasizes the potential exploitation of interpretability to achieve autonomous lifelong learning machines.

Figures (16)

• Figure 1: a Growable online locomotion learning under multicondition (GOLLUM) consists of an interpretable neural control for motor command generation, a dual learning mechanism (primary learning for efficient locomotion learning and supplementary learning for exploiting shared skills), and a neurogenesis for implementing new skills. The interpretable neural control has two interpretation dimensions (column-wise and layer-wise). b In the horizontal/column-wise interpretation, neural columns are created by neurogenesis based on observation and value prediction mismatches. Each encodes a specific behavior/skill, which further includes multiple actions/target configurations. c In the vertical/layer-wise interpretation, four neural modules (sensory preprocessing, internal state, premotor/pattern, and motor/output modules) are stacked to fulfill network functionalities. The sensory preprocessing module is trained supervisedly on observation templates/predictions. The internal state module is precomputed and then fixed during the training. The premotor/pattern module is trained with the supplementary learning to exploit other learned behaviors/skills. The motor/output module is trained with the primary learning to refine/learn behaviors/skills. Combining horizontal and vertical interpretations, each interpretation coordinate thus represents a specific functionality at a specific action of a specific skill. For example, the neuron $\text{C}_\text{9}$ encodes the discrete internal state of the first action of the third skill. The neuron $\text{PM}_\text{10}$ encodes the pattern of the second action of the third skill, and the connection between $\text{PM}_\text{10}$ and an output encodes the output motor command (motor angle) for the second action in the third skill. A video with the neural visualization neurovis is available at https://youtu.be/PxAl___xCT8.
• Figure 2: GOLLUM framework, presented along with the corresponding neural activity signals: feedback ($FB[t]$, where $\theta$ denotes the robot pitch angle), first sensory preprocessing ($I'[t]$, classification score), second sensory preprocessing ($I[t]$, internal state selection), sequential central pattern ($C[t]$, discrete internal state), basis ($B$, smooth internal state), premotor ($PM[t]$, pattern), and output ($M[t]$, motor command). The parameters of the network are summarized in the supplementary document. The signals from the first subnetwork are presented in gray scale, while those from the second subnetworks are in color. Receiving multiple feedback signals at the feedback layer (FB), the first sensory preprocessing layer (I') produces classification signals, which are later selected at the second sensory preprocessing layer (I) based on the activation of the sequential central pattern layer (C). The C layer produces two different sets of discrete internal states: the first group, in gray scale, activates at the start and after the selection signal $I_1[t]$; the second group, in yellowish, activates after the selection signal $I_4[t]$. The basis layer (B) then converts these discrete internal states to smooth internal states, forming the bases for shared action patterns at the premotor layer (PM). The action patterns are then projected to the outputs at the output layer (M, V, and O). The mapping from B to PM is trained by the supplementary learning to activate the proper patterns, while that from PM to M is trained by the primary learning to learn the proper action patterns. Upon encountering new conditions, indicated by a mismatch in value and observation predictions (V and O), the neurogenesis creates new subnetworks for learning new skills. Finally, due to the sparse neuron activation signals, the user/developer can gain insight into the network's processes and learned skills for further modify the results. A video demonstrating the mechanisms of GOLLUM along with the neural visualization neurovis is available at https://youtu.be/PxAl___xCT8.
• Figure 3: MORF hexapod robot employed in this work presented long with its sensors, GOLLUM, the robot interface, and the training process.
• Figure 4: a Feedback independent and b feedback dependent sequential central pattern generator neurons ($C_i$). The former always propagates the activity of the former neuron ($C_{i-1}$) forward to the next ($C_{i+1}$) due to the excitatory connections from $C_i$ to $C_{i+1}$, $C_i$ to $B_i$, and from $B_i$ to $C_{i+1}$. The latter allows the propagation only when the selection input ($I_{i+1}$) is provided due to the excitatory connections from $C_i$ to $C_{i+1}$, from $C_i$ to $I_{i+1}$, and from $I_{i+1}$ to $C_{i+1}$.
• Figure 5: Visualization of the learning rule (Eq. \ref{['eq:rpg']}) applied to the connection weights between two premotor neurons ($PM_{1}$ and $PM_{2}$) and a motor output ($M_1$), where the star denotes the coordinate of the current parameter values, blue dots denote the coordinates of the explored parameters with above-average returns (positive advantages), red dots denote the coordinates of the explored parameters with below-average returns (negative advantages), small grey arrows denote per-sample update gradients, and black arrow denotes the combined parameter update gradient applied to the connection weights. Note that, the size of the dots is proportional to the magnitude of the difference from the average, i.e., the advantages. Therefore, this visualization presents the working process of the learning rule that the update gradient is applied to move the parameters away from the bad explorations with fewer returns and toward the good explorations with higher returns.