Oblique-MERF: Revisiting and Improving MERF for Oblique Photography

TL;DR

Oblique-MERF addresses real-time novel-view synthesis for oblique aerial photography by introducing an adaptive occupancy plane that models occupied space as a sandwiched region between two height-field surfaces and integrates it with volume rendering. It adds a smoothness regularization term that constrains view-direction dependence of specular color to improve extrapolated-view realism. The method achieves about 0.7 dB PSNR improvement and ~40% VRAM reduction, with higher frame rates across most viewpoints, while maintaining rendering fidelity on large-scale scenes. These contributions enable efficient real-time rendering on commodity hardware and suggest avenues for scaling via divide-and-conquer strategies and potential physical rendering integration.

Abstract

Neural implicit fields have established a new paradigm for scene representation, with subsequent work achieving high-quality real-time rendering. However, reconstructing 3D scenes from oblique aerial photography presents unique challenges, such as varying spatial scale distributions and a constrained range of tilt angles, often resulting in high memory consumption and reduced rendering quality at extrapolated viewpoints. In this paper, we enhance MERF to accommodate these data characteristics by introducing an innovative adaptive occupancy plane optimized during the volume rendering process and a smoothness regularization term for view-dependent color to address these issues. Our approach, termed Oblique-MERF, surpasses state-of-the-art real-time methods by approximately 0.7 dB, reduces VRAM usage by about 40%, and achieves higher rendering frame rates with more realistic rendering outcomes across most viewpoints.

Figures (6)

• Figure 1: Overview of our Oblique-MERF pipeline. During training, we introduce a 2D plane to represent the occupied space as a sandwiched region between two height field surfaces. For sampling points on rays, occupancy masks retrieved from the occupancy plane, are used as multipliers in the volume rendering process(section \ref{['section:4.1']}). Additionally, we incorporate a smoothness regularization for view-dependent color to minimize variations in specular color with viewing direction(section \ref{['section:4.2']}). Post-training, spatial occupancy information is directly extracted from the occupancy plane, and corresponding features are stored for real-time rendering(section \ref{['section:4.4']}).
• Figure 2: Camera trajectories from two oblique photography methods.
• Figure 3: In (a), we present training views captured around a building at limited tilt angles. (b) and (c) illustrate the real-time rendering results from a novel extrapolation viewpoint without and with $\mathcal{L}_{\textrm{smooth}}$, respectively. The introduction of the smoothness regularization yields renderings that are smoother and more consistent.
• Figure 4: Comparison on rendering quality for novel view between Oblique-MERF and other methods on the Campus-Oblique and Matrix Cityli2023matrixcity dataset.
• Figure 5: Rendering quality comparison between Oblique-MERF and other MERF variants for test views on the Campus-extra dataset.