Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

Liyou Zhou, Oleg Sinavski, Athanasios Polydoros

TL;DR

SplatGym is presented, an open source neural simulator for training data-driven robotic control policies and broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.

Abstract

The emergence of 3D Gaussian Splatting for fast and high-quality novel view synthesize has opened up the possibility to construct photo-realistic simulations from video for robotic reinforcement learning. While the approach has been demonstrated in several research papers, the software tools used to build such a simulator remain unavailable or proprietary. We present SplatGym, an open source neural simulator for training data-driven robotic control policies. The simulator creates a photorealistic virtual environment from a single video. It supports ego camera view generation, collision detection, and virtual object in-painting. We demonstrate training several visual navigation policies via reinforcement learning. SplatGym represents a notable first step towards an open-source general-purpose neural environment for robotic learning. It broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.

Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

TL;DR

SplatGym is presented, an open source neural simulator for training data-driven robotic control policies and broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.

Abstract

The emergence of 3D Gaussian Splatting for fast and high-quality novel view synthesize has opened up the possibility to construct photo-realistic simulations from video for robotic reinforcement learning. While the approach has been demonstrated in several research papers, the software tools used to build such a simulator remain unavailable or proprietary. We present SplatGym, an open source neural simulator for training data-driven robotic control policies. The simulator creates a photorealistic virtual environment from a single video. It supports ego camera view generation, collision detection, and virtual object in-painting. We demonstrate training several visual navigation policies via reinforcement learning. SplatGym represents a notable first step towards an open-source general-purpose neural environment for robotic learning. It broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.

Paper Structure

This paper contains 25 sections, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Navigation
  4. Neural Radiance Fields for Novel View Synthesis
  5. Neural Simulators for Robotics
  6. Methodology
  7. Problem Statement
  8. Novel View Synthesis
  9. Collision Detection
  10. Implementation: Pre-Processing Pipeline
  11. Video Acquisition and Model Training
  12. Extracting Point Clouds
  13. Octree Representation
  14. Implementation: Software Architecture
  15. Virtual Camera
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Overview of SplatGym, an open-source neural simulator for training data-driven robot control policies. The simulator combines novel view generation and fast collision detection to create a photorealistic simulation environment for reinforcement learning.
  • Figure 2: Novel view images of the same scene and same pose, trained and rendered by NeRF (left) and Gaussian Splatting (right).
  • Figure 3: Pre-processing pipeline of the simulator.
  • Figure 4: Combined octree representation of the garden scene
  • Figure 5: Software architecture of the simulator.
  • ...and 5 more figures