Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments

TL;DR

Experimental results on the Virtual KITTI 2 dataset demonstrate that the proposed approach surpasses state-of-the-art methods across multiple metrics under different noise conditions, and tests using real-world data further validate the effectiveness of the proposed approach.

Abstract

Representing the 3D environment with instance-aware semantic and geometric information is crucial for interaction-aware robots in dynamic environments. Nevertheless, creating such a representation poses challenges due to sensor noise, instance segmentation and tracking errors, and the objects' dynamic motion. This paper introduces a novel particle-based instance-aware semantic occupancy map to tackle these challenges. Particles with an augmented instance state are used to estimate the Probability Hypothesis Density (PHD) of the objects and implicitly model the environment. Utilizing a State-augmented Sequential Monte Carlo PHD (S$^2$MC-PHD) filter, these particles are updated to jointly estimate occupancy status, semantic, and instance IDs, mitigating noise. Additionally, a memory module is adopted to enhance the map's responsiveness to previously observed objects. Experimental results on the Virtual KITTI 2 dataset demonstrate that the proposed approach surpasses state-of-the-art methods across multiple metrics under different noise conditions. Subsequent tests using real-world data further validate the effectiveness of the proposed approach.

Figures (13)

• Figure 1: World model illustration. (a) shows an example scenario with both objects of interests and background objects. The wall and ground belong to the background object with ID 1. The tree belongs to the background object with ID 2. The car and the two robots belong to the objects of interest with ID 3, 4 and 6 respectively, and are dynamic objects. The mailbox with ID 5 is an example static object of interests. Each object is assumed to be composed of a set of points on its surface. The set of the $I$-th object is marked as $\text{X}_I$ and the points within the set share the same color. (b) shows the particles (hollow points) that are used to model the PHD of the points. Particles with different IDs are shown in different colors. Particles with the same ID share the same motion. The particles are stored in voxel subspaces dspMap, which are also used for resampling and occupancy estimation. (c) shows the camera Pinhole model used in this work to formulate the pyramid subspaces dspMap, which are used to distinguish the observed area and occluded area in the continuous space and to accelerate the update process. The green point is a measurement point in a pyramid subspace. The gray area behind the measurement point is occluded. Only a part of the points in $\text{X}_1$ in (a), voxel subspaces in (b), and pyramid subspaces in (c) are shown for clear illustration.
• Figure 2: System structure. The left side shows the input and preprocessing modules, which generate data composed of two parts: transformation matrices and instance point cloud. The generated data, which contains noise, is used in the mapping on the right side. The core of the mapping part is the S$^2$MC-PHD filter.
• Figure 3: Illustration of the updated particles of filters (a) to (c) when a noise observation with a wrong ID is given. With Method IF, the particles' weights at $k-2$ are too small, and the occupancy status of the space is thus falsely treated as free. With Method CF No Forgetting function, the instance ID after $k-1$ is ambiguous.
• Figure 4: Illustration of the memory enhancement. In (a), the blue points are currently observed while the white points are occluded and are aimed to be conjectured. In (b), the memory enhancement structure is shown. The first row shows the process of adding a template in the memory library, which is shown in the middle row. The last row shows the process of matching the particles with a template in the memory library. The green background indicates that the particles or templates correspond to the same semantic label.
• Figure 5: Data structure. Different colors of particles and measurement points represent different instance IDs. The blue object is newly observed. The red rectangle represents the neighbor bounding box area (activation space) of the pixel with a particle in red.