Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Query-based Semantic Gaussian Field for Scene Representation in Reinforcement Learning

Jiaxu Wang, Ziyi Zhang, Qiang Zhang, Jia Li, Jingkai Sun, Mingyuan Sun, Junhao He, Renjing Xu

TL;DR

This work proposes a novel framework that adopts the efficient 3D Gaussian Splatting (3DGS) to learn 3D scene representation for the first time and presents the Query-based Generalizable 3DGS to bridge the 3DGS technique and scene representations with more geometrical awareness than those in NeRFs.

Abstract

Latent scene representation plays a significant role in training reinforcement learning (RL) agents. To obtain good latent vectors describing the scenes, recent works incorporate the 3D-aware latent-conditioned NeRF pipeline into scene representation learning. However, these NeRF-related methods struggle to perceive 3D structural information due to the inefficient dense sampling in volumetric rendering. Moreover, they lack fine-grained semantic information included in their scene representation vectors because they evenly consider free and occupied spaces. Both of them can destroy the performance of downstream RL tasks. To address the above challenges, we propose a novel framework that adopts the efficient 3D Gaussian Splatting (3DGS) to learn 3D scene representation for the first time. In brief, we present the Query-based Generalizable 3DGS to bridge the 3DGS technique and scene representations with more geometrical awareness than those in NeRFs. Moreover, we present the Hierarchical Semantics Encoding to ground the fine-grained semantic features to 3D Gaussians and further distilled to the scene representation vectors. We conduct extensive experiments on two RL platforms including Maniskill2 and Robomimic across 10 different tasks. The results show that our method outperforms the other 5 baselines by a large margin. We achieve the best success rates on 8 tasks and the second-best on the other two tasks.

Query-based Semantic Gaussian Field for Scene Representation in Reinforcement Learning

TL;DR

This work proposes a novel framework that adopts the efficient 3D Gaussian Splatting (3DGS) to learn 3D scene representation for the first time and presents the Query-based Generalizable 3DGS to bridge the 3DGS technique and scene representations with more geometrical awareness than those in NeRFs.

Abstract

Latent scene representation plays a significant role in training reinforcement learning (RL) agents. To obtain good latent vectors describing the scenes, recent works incorporate the 3D-aware latent-conditioned NeRF pipeline into scene representation learning. However, these NeRF-related methods struggle to perceive 3D structural information due to the inefficient dense sampling in volumetric rendering. Moreover, they lack fine-grained semantic information included in their scene representation vectors because they evenly consider free and occupied spaces. Both of them can destroy the performance of downstream RL tasks. To address the above challenges, we propose a novel framework that adopts the efficient 3D Gaussian Splatting (3DGS) to learn 3D scene representation for the first time. In brief, we present the Query-based Generalizable 3DGS to bridge the 3DGS technique and scene representations with more geometrical awareness than those in NeRFs. Moreover, we present the Hierarchical Semantics Encoding to ground the fine-grained semantic features to 3D Gaussians and further distilled to the scene representation vectors. We conduct extensive experiments on two RL platforms including Maniskill2 and Robomimic across 10 different tasks. The results show that our method outperforms the other 5 baselines by a large margin. We achieve the best success rates on 8 tasks and the second-best on the other two tasks.
Paper Structure (17 sections, 12 equations, 3 figures, 2 tables)

This paper contains 17 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Scene Representations
  4. Representing 3D Scene as Latent Vector
  5. Learnable State Representations for Reinforcement Learning.
  6. Background
  7. Reinforcement Learning
  8. 3D Gaussian Splatting
  9. Method
  10. Hierarchical Semantics Encoding
  11. Query-based Generalizable Feature Splatting
  12. Experiments
  13. Experimental Settings
  14. Implementation details
  15. Results Discussion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The main pipeline of the proposed approach. The left panel is the Query-based Generalizable Gaussian Splatting, which uses points to query the scene vector to obtain geometry parameters and features and then render novel views via Gaussian Splatting. The right panel is the Hierarchical Semantics Encoding that hierarchically grounds part-level semantic features into the Gaussian field to distill the scene vector.
  • Figure 2: Learning Curves Comparisons of ours and the other 5 baselines.
  • Figure 3: The training curves of the ablation studies: (a) the ablation of different settings of semantic encoding, (b) the ablation of the different numbers of input images.