Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Photorealistic Robotic Simulation using Unreal Engine 5 for Agricultural Applications

Xingjian Li, Lirong Xiang

TL;DR

The paper addresses the need for photorealistic, controllable simulation environments to accelerate development of agricultural robotics without the overhead of real-field experiments. It proposes a Unreal Engine 5 (UE5) based simulator tightly integrated with ROS to generate realistic plant imagery (RGB, depth, segmentation) and support multi-robot planning with two UR10 arms. Key contributions include a UE5-driven photorealistic environment leveraging Lumen/Nanite and Quixel Megascans, a ROS–UE5 integration for joint-state synchronization and image capture, and a quantitative assessment showing high trajectory fidelity (e.g., average one-second error of $0.021$ mm) alongside realistic rendering. The work advances synthetic-data generation for agricultural perception and planning and lays groundwork for future enhancements such as gripper manipulation, mobile platforms, and real-world validation with expanded plant assets.

Abstract

This work presents a new robotics simulation environment built upon Unreal Engine 5 (UE5) for agricultural image data generation. The simulation utilizes the state-of-the-art real-time rendering engine to provide realistic plant images which are often used in agricultural applications. This study showcases the rendering accuracy of UE5 in comparison to existing tools and assesses its positional accuracy when integrated with Robot Operating Systems (ROS). The results indicate that UE5 achieves an impressive average distance error of 0.021mm when compared to predetermined setpoints in a multi-robot setup involving two UR10 arms.

Photorealistic Robotic Simulation using Unreal Engine 5 for Agricultural Applications

TL;DR

The paper addresses the need for photorealistic, controllable simulation environments to accelerate development of agricultural robotics without the overhead of real-field experiments. It proposes a Unreal Engine 5 (UE5) based simulator tightly integrated with ROS to generate realistic plant imagery (RGB, depth, segmentation) and support multi-robot planning with two UR10 arms. Key contributions include a UE5-driven photorealistic environment leveraging Lumen/Nanite and Quixel Megascans, a ROS–UE5 integration for joint-state synchronization and image capture, and a quantitative assessment showing high trajectory fidelity (e.g., average one-second error of mm) alongside realistic rendering. The work advances synthetic-data generation for agricultural perception and planning and lays groundwork for future enhancements such as gripper manipulation, mobile platforms, and real-world validation with expanded plant assets.

Abstract

This work presents a new robotics simulation environment built upon Unreal Engine 5 (UE5) for agricultural image data generation. The simulation utilizes the state-of-the-art real-time rendering engine to provide realistic plant images which are often used in agricultural applications. This study showcases the rendering accuracy of UE5 in comparison to existing tools and assesses its positional accuracy when integrated with Robot Operating Systems (ROS). The results indicate that UE5 achieves an impressive average distance error of 0.021mm when compared to predetermined setpoints in a multi-robot setup involving two UR10 arms.
Paper Structure (11 sections, 4 figures)

This paper contains 11 sections, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. SIMULATION SETUP
  4. UE5 Environment
  5. UE5 & ROS Robots
  6. Communication
  7. EXPERIMENT
  8. RESULTS
  9. Synthetic Images
  10. Simulation Accuracy
  11. CONCLUSIONS AND FUTURE WORK

Figures (4)

  • Figure 1: Unreal Engine 5 Environment
  • Figure 2: Rendered RGB and segmentation image comparison. The segmentation colors for a) are determined by the base color of the material of the copied room as described in Section \ref{['sec:UE5 Environment']}. Notice the lack of details in b)'s segmentation compared to a)'s segmentation, where the plugin removes the transparency of the material by replacing it with new material.
  • Figure 3: Point clouds from UE5 RGBD Images displayed in RViz during the simulation. The point cloud below the plant is circular because it is a cylinder in UE5.
  • Figure 4: End effector trajectories. Left: world positions of the robot 1 end effector for UE5 and ROS of one trajectory with the highest same-time distance error (0.77m at one point). Right: the 1-second window of UE5 distance error after ROS has reached one of 120 setpoints.