Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigation

TL;DR

Seq-DeepIPC addresses end-to-end navigation for legged robots in mixed terrains by unifying sequential perception with planning and control. The model ingests a short sequence of RGB-D frames, performs dual-head segmentation and depth estimation, projects features into a BEV, and uses a GRU-based planner to output future waypoints and control commands. By replacing heavier encoders with EfficientNet-B0 and using GNSS-based bearing instead of IMU, the system achieves edge-friendly deployment and improved heading stability in open areas. Experiments on an extended campus dataset with road and grass show that temporal inputs and multi-task perception yield more stable perception, accurate waypoint predictions, and smoother control, while GNSS-based heading still struggles near tall buildings, suggesting benefits from sensor fusion.

Abstract

We present Seq-DeepIPC, a sequential end-to-end perception-to-control model for legged robot navigation in realworld environments. Seq-DeepIPC advances intelligent sensing for autonomous legged navigation by tightly integrating multi-modal perception (RGB-D + GNSS) with temporal fusion and control. The model jointly predicts semantic segmentation and depth estimation, giving richer spatial features for planning and control. For efficient deployment on edge devices, we use EfficientNet-B0 as the encoder, reducing computation while maintaining accuracy. Heading estimation is simplified by removing the noisy IMU and instead computing the bearing angle directly from consecutive GNSS positions. We collected a larger and more diverse dataset that includes both road and grass terrains, and validated Seq-DeepIPC on a robot dog. Comparative and ablation studies show that sequential inputs improve perception and control in our models, while other baselines do not benefit. Seq-DeepIPC achieves competitive or better results with reasonable model size; although GNSS-only heading is less reliable near tall buildings, it is robust in open areas. Overall, Seq-DeepIPC extends end-to-end navigation beyond wheeled robots to more versatile and temporally-aware systems. To support future research, we will release the codes to our GitHub repository at https://github.com/oskarnatan/Seq-DeepIPC.

Figures (6)

• Figure 1: Overview of the proposed Seq-DeepIPC architecture. (1) Perception part: Sequential RGB inputs are processed by an EfficientNet-B0 encoder, producing latent features that drive two prediction heads for semantic segmentation and depth estimation. The ground-truth depth maps are combined with predicted segmentation to generate BEV projections, which are further encoded by a second EfficientNet-B0 into BEV latent features. (2) Planning and Control part: The fused RGB and BEV latent features, together with transformed route points and bearing angle, are processed by a GRU to capture temporal dependencies. The resulting features drive two complementary control pathways: (a) PID controllers, which use predicted waypoints to estimate control signals, and (b) command-specific MLP controllers, which directly map the GRU latent space to $(x,y,\theta)$ controls. The blended control policy regulates position and orientation for the legged robot.
• Figure 2: GNSS-based bearing estimation (orange) vs 9-axis IMU with EKF-based bearing estimation (purple).
• Figure 3: The experiment area. Red: Old DeepIPC dataset coverage area. Blue: extended coverage area used for all models in this experiment. Yellow hollow circles represent a route that consists of start, finish, and a set of route points. (https://goo.gl/maps/9rXobdhP3VYdjXn48)
• Figure 4: Sensor placement on the legged robot (Unitree Go2).
• Figure 5: Qualitative results of Seq-DeepIPC deployment on a Unitree legged robot. Each row corresponds to an observation set, showing representative outputs. Negative x-pos and orient. controls mean to the left, while positive means to the right. (a) Successful road traversal, (b) successful grass (stairs) traversal, (c) failure case near tall buildings, as the model fails to predict waypoints correctly due to misplaced route points.