Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots

Abstract

Panoramic imagery provides holistic 360° visual coverage for perception in quadruped robots. However, existing occupancy prediction methods are mainly designed for wheeled autonomous driving and rely heavily on RGB cues, limiting their robustness in complex environments. To bridge this gap, (1) we present PanoMMOcc, the first real-world panoramic multimodal occupancy dataset for quadruped robots, featuring four sensing modalities across diverse scenes. (2) We propose a panoramic multimodal occupancy perception framework, VoxelHound, tailored for legged mobility and spherical imaging. Specifically, we design (i) a Vertical Jitter Compensation (VJC) module to mitigate severe viewpoint perturbations caused by body pitch and roll during mobility, enabling more consistent spatial reasoning, and (ii) an effective Multimodal Information Prompt Fusion (MIPF) module that jointly leverages panoramic visual cues and auxiliary modalities to enhance volumetric occupancy prediction. (3) We establish a benchmark based on PanoMMOcc and provide detailed data analysis to enable systematic evaluation of perception methods under challenging embodied scenarios. Extensive experiments demonstrate that VoxelHound achieves state-of-the-art performance on PanoMMOcc (+4.16%} in mIoU). The dataset and code will be publicly released to facilitate future research on panoramic multimodal 3D perception for embodied robotic systems at https://github.com/SXDR/PanoMMOcc, along with the calibration tools released at https://github.com/losehu/CameraLiDAR-Calib.

Figures (15)

• Figure 1: Comparison between Vehicle-based and Quadruped-based Occupancy Perception. (a) Vehicle setting vs. (b) Quadruped setting.
• Figure 2: Overview of the proposed dataset across six representative scenes. We visualize a cropped region of the panoramic ERP image for clarity. The dataset covers diverse scenes and challenging conditions for multimodal occupancy perception.
• Figure 3: Features of the proposed PanoMMOcc dataset. (a) Quadruped-based multimodal data acquisition platform. (b) Semantic class distribution of the annotated occupancy labels.
• Figure 4: Overview of the proposed VoxelHound. The camera branch receives panoramic RGB, thermal, and polarization images, while the LiDAR branch processes raw point clouds. Each modality is encoded by an independent encoder, and the extracted features are projected into the BEV space for multimodal fusion.
• Figure 5: Overview of the proposed modules. a) Vertical Jitter Compensation (VJC) module mitigates image distortions caused by quadruped locomotion. b) Multimodal Information Prompt Fusion (MIPF) module for effective multimodal feature fusion.