A Neural Field-Based Approach for View Computation & Data Exploration in 3D Urban Environments

TL;DR

This work addresses the challenge of extracting insights from large 3D urban datasets under occlusion and interaction bottlenecks by modeling views as a vector field over viewpoints and learning a neural field-based implicit representation $S: \mathcal{V} \to \mathbb{R}^k$. It introduces a differentiable architecture $F_\Theta$ with high-frequency embeddings to approximate the field, supporting direct queries for view-dependent thematic distributions and inverse queries to locate viewpoints matching desired patterns. The authors demonstrate three urban-use cases (facade visibility, pattern-driven view discovery, and building-centric visibility) on a Manhattan dataset, with quantitative accuracy and speed benchmarks (e.g., ~$4{,}000$ views/s per instance on a $2.4$ MB network) and expert feedback, and provide an open-source codebase. The approach enables fast, scalable, and interactive exploration of physical and thematic urban data, offering practical tools for planning, design, and environmental assessment while outlining avenues for extension to rural settings and additional thematic layers.

Abstract

Despite the growing availability of 3D urban datasets, extracting insights remains challenging due to computational bottlenecks and the complexity of interacting with data. In fact, the intricate geometry of 3D urban environments results in high degrees of occlusion and requires extensive manual viewpoint adjustments that make large-scale exploration inefficient. To address this, we propose a view-based approach for 3D data exploration, where a vector field encodes views from the environment. To support this approach, we introduce a neural field-based method that constructs an efficient implicit representation of 3D environments. This representation enables both faster direct queries, which consist of the computation of view assessment indices, and inverse queries, which help avoid occlusion and facilitate the search for views that match desired data patterns. Our approach supports key urban analysis tasks such as visibility assessments, solar exposure evaluation, and assessing the visual impact of new developments. We validate our method through quantitative experiments, case studies informed by real-world urban challenges, and feedback from domain experts. Results show its effectiveness in finding desirable viewpoints, analyzing building facade visibility, and evaluating views from outdoor spaces. Code and data are publicly available at https://urbantk.org/neural-3d.

Figures (6)

• Figure 1: Our proposed visibility-based exploration is enable by two types of queries supported by a machine learning model. Direct queries (top), computing a thematic vector from spatial positions. Inverse queries (bottom), searching spatial positions satisfying given target view constraints.
• Figure 2: The interface includes four main components: Control query panel with a PCP for visualizing ground truth visibility distributions. Users can select distribution type (semantics/perception), brush axes to define query constraints, and specify a hyperplane and number of locations to generate new camera views using the model. Latent map view shows model predictions (purple) and generated views (green) in latent space; users can change projection, zoom/pan, and brush to highlight points. Gallery view displays the model-generated camera views associated with brushed green points (from ) in a scrollable panel, and allows selecting a view to update the 3D map view. The 3D scene supports navigation, with options to compute visibility of elements like sky, tree, water from building facades, visualize it on facade tiles, with a reference color legend.
• Figure 3: Three scenarios for the thematic distribution of building materials. On the left side we display aerial views of the scene. On the right we show the cumulative view error of our model with respect to the regions of the scene. In all three scenarios the model predicts the view thematic composition with less than $10\%$ error for $\approx 80\%$ of regions in the scene.
• Figure 4: RMSE of our model compared to baseline small-footprint ML models across varying training set sizes.
• Figure 5: Inverse query for views towards trees and sky located on a street. We identify viewpoint clusters using the latent representation of each location generated by our model. In , and we show a selection of views found through the inverse query.