Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UnrealCV: Connecting Computer Vision to Unreal Engine

Weichao Qiu, Alan Yuille

TL;DR

The paper introduces UnrealCV, an open-source plugin that extends Unreal Engine 4 with a server–client architecture to access internal game data, control agents, and modify world properties, enabling synthetic data generation with rich ground-truth and integration with deep learning workflows. It argues that virtual worlds offer valuable, controllable environments for training, testing, and evaluating computer-vision algorithms, and demonstrates two applications: synthetic dataset generation and cross-view evaluation of a Faster-RCNN model within a controllable virtual scene. The authors detail the architecture (UE4 plugin server and external client), the vget/vset command protocol, and the system's extensibility and cross-language compatibility, highlighting both the potential and current limitations of synthetic-to-real transfer. They position UnrealCV as a bridge between the game/VR ecosystem and CV research, with open-source availability to foster community-driven expansion and richer virtual content.

Abstract

Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (http://unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.

UnrealCV: Connecting Computer Vision to Unreal Engine

TL;DR

The paper introduces UnrealCV, an open-source plugin that extends Unreal Engine 4 with a server–client architecture to access internal game data, control agents, and modify world properties, enabling synthetic data generation with rich ground-truth and integration with deep learning workflows. It argues that virtual worlds offer valuable, controllable environments for training, testing, and evaluating computer-vision algorithms, and demonstrates two applications: synthetic dataset generation and cross-view evaluation of a Faster-RCNN model within a controllable virtual scene. The authors detail the architecture (UE4 plugin server and external client), the vget/vset command protocol, and the system's extensibility and cross-language compatibility, highlighting both the potential and current limitations of synthetic-to-real transfer. They position UnrealCV as a bridge between the game/VR ecosystem and CV research, with open-source availability to foster community-driven expansion and richer virtual content.

Abstract

Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (http://unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.

Paper Structure

This paper contains 6 sections, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unreal Engine
  4. UnrealCV
  5. Applications
  6. Conclusion

Figures (4)

  • Figure 1: A synthetic image and its ground truth generated using UnrealCV. The virtual room is from technical demo RealisticRendering, built by Epic Games. From left to right are the synthetic image, object instance mask, depth, surface normal
  • Figure 2: Images produced by UE4, (a)(b) An architectural visualization and an urban city scene from Unreal Engine marketplace. (c) An open-source outdoor scene KiteRunner. (d) A digital human from the game Hellblade, shown in the conference GDC2016
  • Figure 3: The UnrealCV server is an UE4 plugin embedded into a game during compilation. An external program uses the UnrealCV client to communicate with the game.
  • Figure 4: Images with different camera height and different sofa color.