RFUniverse: A Multiphysics Simulation Platform for Embodied AI

TL;DR

RFUniverse introduces a multiphysics-based simulation platform for embodied AI that integrates aerodynamics, hydrodynamics, and thermodynamics with rigid/multi-body dynamics in a Unity-backed client-server framework. The system provides physics-based rendering, multi-modal sensing, and lightweight planning tools, enabling RL and planning benchmarks on tasks that involve coupling effects. The paper presents three RL tasks (physics-based cutting, water pushing, towel catching) and a planning-based butter pushing to illustrate the platform's capabilities, alongside fidelity verification against real-world phenomena. The work demonstrates that a real-time, multiphysics simulator can broaden the task scope for embodied AI and serve as an effective testbed for sim-to-real research, while outlining future work to add more solvers and scenarios.

Abstract

Multiphysics phenomena, the coupling effects involving different aspects of physics laws, are pervasive in the real world and can often be encountered when performing everyday household tasks. Intelligent agents which seek to assist or replace human laborers will need to learn to cope with such phenomena in household task settings. To equip the agents with such kind of abilities, the research community needs a simulation environment, which will have the capability to serve as the testbed for the training process of these intelligent agents, to have the ability to support multiphysics coupling effects. Though many mature simulation software for multiphysics simulation have been adopted in industrial production, such techniques have not been applied to robot learning or embodied AI research. To bridge the gap, we propose a novel simulation environment named RFUniverse. This simulator can not only compute rigid and multi-body dynamics, but also multiphysics coupling effects commonly observed in daily life, such as air-solid interaction, fluid-solid interaction, and heat transfer. Because of the unique multiphysics capacities of this simulator, we can benchmark tasks that involve complex dynamics due to multiphysics coupling effects in a simulation environment before deploying to the real world. RFUniverse provides multiple interfaces to let the users interact with the virtual world in various ways, which is helpful and essential for learning, planning, and control. We benchmark three tasks with reinforcement learning, including food cutting, water pushing, and towel catching. We also evaluate butter pushing with a classic planning-control paradigm. This simulator offers an enhancement of physics simulation in terms of the computation of multiphysics coupling effects.

Figures (8)

• Figure 1: Tasks in RFUniverse: Standard tasks such as (a) Navigation (b) Pick and place, and tasks involving multiphysics interaction such as (c) Physics-based cutting; (d) Water pushing; (e) Towel catching; (f) Butter pushing.
• Figure 2: The framework of RFUniverse.
• Figure 3: The VR interface with HTC Vive and Noitom Glove.
• Figure 4: (a)(b): Refractive effect in real world and in RFUniverse. Two wine glasses with gray code behind them. Here we mainly compare the refraction effects because it can reveal the physics fidelity of the rendering system; (c)-(e): different indoor lighting conditions; (f)-(h): Liquid in refraction rendering effects under basic rendering, 'photorealistic' rendering claimed by other simulation environments, and ray-tracing (physics-based) rendering.
• Figure 5: Multi-modal Sensor. (a) Visual sensor: RGB, Instance mask, Perfect depth and Normal map; (b) IR-based depth sensor; (c) DIGIT Tactile Sensor with tactile image. We also compare our implementation with real-world DIGIT sensor and official implementation in Pybullet to show that the gap between DIGIT in real-world in RFUniverse is significantly smaller. For the real-world one, we take the image in the same setting.; (d) Force/Torque Sensor.