Backpropagation through Soft Body: Investigating Information Processing in Brain-Body Coupling Systems

TL;DR

Backpropagation through soft body (BPTSB) investigates how information processing is distributed between brain and body in brain–body–environment coupling using a differentiable mass-spring-damper (MSDN) body and two brain models (a feed-forward neural network variant and a sine-wave generator). The authors train the entire brain–body system end-to-end, quantify functional division via MNIST classification and time-series emulation, and demonstrate that portions of brain functionality can be embedded into the body to achieve autonomous closed-loop control via a learned feedback layer. They reveal reciprocal brain–body dynamics, show that the body can absorb output variability to aid recognition, and show that higher-order nonlinearities emerge through joint optimization, with memory properties distributed across body states. The work advances embodied computation and suggests practical paths for efficient brain–body co-design in soft robotics and adaptive control, leveraging physical reservoir computing principles and differentiable physics.

Abstract

Animals achieve sophisticated behavioral control through dynamic coupling of the brain, body, and environment. Accordingly, the co-design approach, in which both the controllers and the physical properties are optimized simultaneously, has been suggested for generating refined agents without designing each component separately. In this study, we aim to reveal how the function of the information processing is distributed between brains and bodies while applying the co-design approach. Using a framework called ``backpropagation through soft body," we developed agents to perform specified tasks and analyzed their mechanisms. The tasks included classification and corresponding behavioral association, nonlinear dynamical system emulation, and autonomous behavioral generation. In each case, our analyses revealed reciprocal relationships between the brains and bodies. In addition, we show that optimized brain functionalities can be embedded into bodies using physical reservoir computing techniques. Our results pave the way for efficient designs of brain--body coupling systems.

Figures (12)

• Figure 1: Backpropagation through soft body. The system is composed of a brain and a body. The brain is a feed-forward neural network (FNN) or a sine wave generator (SWG), and its output $\bm{y}_{\text{br}}$ corresponds to the input to the body. The body, a mass-spring-damper network (MSDN), produces physical movements through spring and external forces. The input $\bm{u}$ is conveyed to the brain, and the output $\bm{y}$ is generated via information processing and dynamics. While calculating the error $E$ with a loss function $\mathcal{L}$ based on the generated output $\bm{y}$ and the target output $\bm{y}_{\text{tgt}}$, the parameters are updated by propagating errors backward through the entire system. The parameters in red (the weights of the FNN $W$ and $b$, the amplitude $A$ and phase $\phi$ of the SWG, and the spring constant $k$, damping coefficient $d$, and rest length $l$ of the springs) are trainable.
• Figure 2: An MNIST classification and drawing task. (a) The input, an MNIST image, is conveyed to the brain, and the brain transforms the input information into the initial positions of the movable mass points (blue points; exclude the central mass point (CMP)). The trajectory of the CMP becomes the output. The trajectory is color-matched with the elapsed time. The optimized system draws the shape of the input label number (see Supplementary Video 1). (b) The errors calculated based on the drawn trajectories of the trained system with each configuration. The table below shows the actual trajectories and the errors of the best samples in their interactions with a multiple-circle body ($N_{\text{mov}}=17$). (c) The outputs of the brain (the initial positions of the movable mass points) and the body (the trajectory of the CMP) are compressed into two dimensions using principal component analysis (PCA). Each color corresponds to the input label. The more separated the dots on the basis of color, the stronger the label classification. (d) Color coding is performed based on the trajectory drawn via transient dynamics. Among the initial positions of the movable mass points, which are determined by the brain, one mass point (yellow) is displaced, and the dynamics are then measured. The colors correspond to the labels with the least error between the actual and target trajectories. For instance, the brown area indicates that the shape of the trajectory is most similar to a five. Two examples in which one mass point is displaced from its initial position for the label "five" are displayed. As the point is shifted to the upper right, the trajectory changes to six in the top figure. However, in the lower part of the figure, the trajectory is almost constant. (e) The color represents the degree of simplicity of the label maps, which means that the trajectory becomes more sensitive to the initial position as the value decreases.
• Figure 3: Time-series emulation task. (a) The input is an independent and identically distributed uniform random value, and the output corresponds to the $y$-axis position of the CMP. The target output is calculated via the nonlinear transformation of the past input and output values. The one-dimensional input $u(t)$ is transformed by the brain into external forces and then sequentially conveyed to the movable mass points (except the CMP). While the body behaves in accordance with the external forces, the $y$-coordinate of the CMP corresponds to the system output $y(t+1)$ (Supplementary Video 2). (b) The input, target, and output time-series. The table below lists the reconstruction errors of each system, with the background color corresponding to the color of the time-series plots. (c) The left side represents the information processing capacity (IPC) values of the system output, which are arranged from left to right according to training epochs. The rightmost bar represents the IPC value of the target time-series. The middle figures represent the IPC of each movable mass point (one dimension each in the $x$ and $y$ axes). These are the results of the system that combines a double-circle structured body with an LIL or an MLP, where the mass point ID is given in a clockwise direction, as shown in the upper right. The degrees of the orthogonal polynomials and the delayed time steps of the input are given as $n$ and $\Delta t$, respectively, in $\mathcal{F}_n\left(u(t-\Delta t)\right)$, and they are equivalent to the colors and intensity of the bars. In particular, the blue parts correspond to linear profiles. The total capacity reached the rank of one in all cases.
• Figure 4: Closed-loop control by feedback layer. (a) A feedback layer (FL) is added to an open-loop system composed of an SWG and an MSDN, and it is then trained in an open loop. After this training, a closed loop is formed by designating the FL as the brain instead of the SWG. The lower row shows the behaviors in an open loop and a closed loop (Supplementary Video 3 and 4). (b) The robustness is tested by applying perturbations to the positions of the movable mass points (shifting the gray points to the blue points) and checking if the system returns to the desired behavior afterwards. In the Lissajous curve drawing task, the upper right figures show the trajectories of the systems; one could return and the other could not return to the trained trajectory. In the locomotion task, the locomotion speeds of the systems are displayed in the lower right. A perturbation was applied to the system according to the timing of the orange background color. (c) It shows the rate at which the system returns to the trained behavior based on the magnitude of the perturbation. The standard deviation when generating a perturbation is parameterized, which corresponds to the value of the horizontal axis.
• Figure 5: The arrangement of the mass points.