360CityGML: Realistic and Interactive Urban Visualization System Integrating CityGML Model and 360° Videos

TL;DR

This work tackles the realism gap in CityGML-based urban visualization by integrating 360° walkthrough videos as dynamic textures on CityGML geometries. It introduces a street-level offline alignment using Visual SLAM to estimate a 7-DOF local-to-global transform and optimizes it with CMA-ES, aligning video frames to the CityGML model based on building-region discrepancies from semantic segmentation. Online, an avatar-driven view-selection strategy, constrained to camera trajectories and smoothed via quaternion SLERP, selects appropriate 360° frames to texture-projection, enabling seamless, photorealistic pedestrian navigation. The system is demonstrated in Akihabara with flood-risk and daylight analyses and is validated by a user study showing enhanced presence and realism with textured views, highlighting practical impact for urban planning and risk communication.

Abstract

We introduce a novel urban visualization system that integrates 3D urban model (CityGML) and 360° walkthrough videos. By aligning the videos with the model and dynamically projecting relevant video frames onto the geometries, our system creates photorealistic urban visualizations, allowing users to intuitively interpret geospatial data from a pedestrian view.

Figures (11)

• Figure 1: Comparison between the conventional static texture mapping of each face boorboor2024submerse (top), and our dynamic texture projection of 360° video frames (bottom). Our approach faithfully reproduces the real-world appearance to a much greater extent.
• Figure 2: An overview of 360CityGML. Our system first aligns each frame of the 360° video with the CityGML model in an offline process. In the online system, a user can virtually walk through the environment by controlling their avatar. Based on the avatar's location and its movement direction, the system automatically selects the most suitable 360° video frame and its spherical orientation as the background. This frame is then directly projected onto the 3D urban model as a texture, delivering a photorealistic visual experience. Finally, the textured models are overlaid with geospatial data, offering highly photorealistic and 3D-aware urban visualizations.
• Figure 3: An overview of the formulation of our street-level alignment. First, we apply Visual SLAM sumikura2019openvslam to each 360° video to obtain the camera trajectory in local coordinates systems. Next, we perform a local-to-global transformation using the position of the camera's starting point $\bm{v}_s$, the position of the end point $\bm{v}_e$, and the rotation angle $\lambda$ around the gravity axis. This transformation aligns the video frames with the global CityGML coordinate system.
• Figure 4: An overview of our objective function. For a sampled subset of video frame, we calculate the total number of discrepant pixels by comparing the building regions in the model projection to those in the video frames extracted by semantic segmentation cheng2022masked. We then optimize the parameters ($\bm{v}_s$, $\bm{v}_e$, $\lambda$) to minimize this discrepancy.
• Figure 5: Overviews of our view-selection strategy. (a) For viewpoint selection, our system calculate the distance $d \in \mathbb{R}$, representing how far the avatar has moved along the trajectory direction. We shift the viewpoint to adjacent frames when the avatar exceeds the threshold distance $\alpha \in \mathbb{R}$. (b) For view orientation selection, we first compute $\bm{\theta}_\mathrm{avatar} \in \mathbb{R}^3$, which denotes the camera orientation that faces the avatar from the current viewpoint. We then apply a temporal moving average to smooth orientation changes relative to the trajectory direction $\bm{\theta}_i \in \mathbb{R}^3$, effectively reducing unwanted oscillations.