DiffTex: Differentiable Texturing for Architectural Proxy Models

TL;DR

DiffTex tackles texture recovery for architectural proxy models by learning texel-level textures from unordered, registered photographs using differentiable rendering. It establishes a per-texel UV-to-image correspondence and represents texel color as a weighted combination of input pixels, optimized under photometric and perspective losses $L_{Render}$, $L_{Persp}$, and $L_{Para}$, with data-quality refinements and a coarse-to-fine pipeline. The approach yields robust, seam-free textures on diverse facades, outperforming prior methods on synthetic and real scenes in terms of perceptual quality (SSIM, LPIPS) and perspective consistency. The method enables photorealistic, efficient texturing of architectural proxies suitable for real-time rendering and visualization.

Abstract

Simplified proxy models are commonly used to represent architectural structures, reducing storage requirements and enabling real-time rendering. However, the geometric simplifications inherent in proxies result in a loss of fine color and geometric details, making it essential for textures to compensate for the loss. Preserving the rich texture information from the original dense architectural reconstructions remains a daunting task, particularly when working with unordered RGB photographs. We propose an automated method for generating realistic texture maps for architectural proxy models at the texel level from an unordered collection of registered photographs. Our approach establishes correspondences between texels on a UV map and pixels in the input images, with each texel's color computed as a weighted blend of associated pixel values. Using differentiable rendering, we optimize blending parameters to ensure photometric and perspective consistency, while maintaining seamless texture coherence. Experimental results demonstrate the effectiveness and robustness of our method across diverse architectural models and varying photographic conditions, enabling the creation of high-quality textures that preserve visual fidelity and structural detail.

Figures (10)

• Figure 1: The differential render renders views to compute a loss, which propagates back to optimize the parameters.
• Figure 2: The photo $\mathcal{I}_k^i$ with binary activation mask $a_k^i$ (left), and the mapped photo $\widehat{\mathcal{I}}_k^i$ with mapped $\widehat{a}_k^i$ (right) against proxy polygon $\mathcal{P}_i$ of Hisense.
• Figure 3: The initial facade texture maps, and the initial texture maps optimized with $L_{Render}$, $L_{Persp}$, $L_{Para}$, $L_{Render} + L_{Persp} + L_{Para}$ on HighRise example are visualized from left to right. For each facade texture, a texel source map visualizing the photo source (with dominant $\mathcal{W}$ value) of all texels is placed on its right. Each input photo is represented by a unique, randomly assigned color.
• Figure 4: Comparison with LTBC, 2DGS, PBO, TwinTex and ours on facades of two virtual buildings. The texel-wise differences compared with the GT maps are visualized on the right. The colors denote error values from low (white) to high (dark red).
• Figure 5: Comparison of texturing results with RC, LTBC, PBO, TwinTex, and ours on real-world buildings.