Virtual Fieldwork in Immersive Environments using Game Engines

TL;DR

The paper presents a Virtual Fieldwork Tool (VFT) built in Unreal Engine 5.3 and deployed in the ARENA2 immersive dome to support collaborative, immersive analysis of seafloor datasets. It integrates georeferenced photogrammetry and hydroacoustic models via Cesium for Unreal and multi-display UE rendering, providing intuitive measurement tools such as distance, strike/dip, and clipping with JSON export. Three real-world use cases (Kolumbo, Mothra, Niua South) demonstrate the system's capabilities across scales and data origins, highlighting benefits for interpretation and collaboration as well as limitations in data variability and workflow integration. The work argues that immersive geoscience visualization can complement traditional GIS workflows and outlines concrete future enhancements for broader adoption and education.

Abstract

Fieldwork still is the first and foremost source of insight in many disciplines of the geosciences. Virtual fieldwork is an approach meant to enable scientists trained in fieldwork to apply these skills to a virtual representation of outcrops that are inaccessible to humans e.g. due to being located on the seafloor. For this purpose we develop a virtual fieldwork software in the game engine and 3D creation tool Unreal Engine. This software is developed specifically for a large, spatially immersive environment as well as virtual reality using head-mounted displays. It contains multiple options for quantitative measurements of visualized 3D model data. We visualize three distinct real-world datasets gathered by different photogrammetric and bathymetric methods as use cases and gather initial feedback from domain experts.

Figures (20)

• Figure 1: Interactively analyzing a seafloor bathymetry 3D model inside the ARENA2 at GEOMAR Helmholtz Centre for Ocean Research Kiel.
• Figure 2: A cut-away sketch of the ARENA2. Shown is the roughly $7 m \times 7 m \times 6 m$ enclosure which is fully opaque in reality. Also visible is the scaffolding on which the free-hanging dome structure is suspended at a 21° angle.
• Figure 3: An overview of several of the devices we are using in the ARENA2. Visible in the background is the inside surface of the projection dome. 1: One of the five BARCO F50 WQXGA projectors. 2: One of the four OptiTrack motion tracking cameras, recognizable by the blue glowing ring. 3: Stereo 3D shutter glasses and an Xbox controller equipped with tracking markers. 4: An HTC Vive VR headset with the accompanying controllers. 5: One of two HTC Vive base stations ("lighthouse") used to track the position of the Vive headset and controllers.
• Figure 5: Our photogrammetry preprocessing workflow for adding models to our Unreal Engine VFT. Images are processed in the respective photogrammetry software and a mesh is created which receives a texture calculated also from the images. This textured mesh can then either be exported directly as Cesium tileset or to an FBX file and imported as static mesh asset to UE.
• Figure 6: The starting level in which a user is placed initially when starting the VFT. The "portals" labeled with the name of the location teleport the user to the respective model. The compass widget visible in the lower left part of the image can be hidden.