2D Neural Fields with Learned Discontinuities

TL;DR

This work tackles the challenge that continuous neural fields blur sharp image discontinuities. It introduces a discontinuous 2D neural field built on a triangle mesh, where edge discontinuities are learned as continuous parameters associated with every mesh edge and refined via a rounding step. The approach yields substantial gains over continuous baselines such as InstantNGP in denoising ($>5$ dB) and super-resolution ($>10$ dB), and achieves more faithful discontinuity recovery than Mumford–Shah-based methods. It enables edge-preserving reconstruction, vectorization of art and depth maps, and robust handling of complex drawings, with potential for broader application to diffusion-curve geometry and depth segmentation.

Abstract

Effective representation of 2D images is fundamental in digital image processing, where traditional methods like raster and vector graphics struggle with sharpness and textural complexity respectively. Current neural fields offer high-fidelity and resolution independence but require predefined meshes with known discontinuities, restricting their utility. We observe that by treating all mesh edges as potential discontinuities, we can represent the magnitude of discontinuities with continuous variables and optimize. Based on this observation, we introduce a novel discontinuous neural field model that jointly approximate the target image and recovers discontinuities. Through systematic evaluations, our neural field demonstrates superior performance in denoising and super-resolution tasks compared to InstantNGP, achieving improvements of over 5dB and 10dB, respectively. Our model also outperforms Mumford-Shah-based methods in accurately capturing discontinuities, with Chamfer distances 3.5x closer to the ground truth. Additionally, our approach shows remarkable capability in handling complex artistic drawings and natural images.

Figures (18)

• Figure 1: Discontinuity-aware 2D neural field belhe23 requires accurate 2D discontinuities as input. In their example of denoising 3D renderings, all types of discontinuities may not always be available. False negatives caused by sharp texture and refracted geometries lead to blurs.
• Figure 2: (a,b) Diffusion curves define an example harmonic function field with sharp discontinuities orzan08. (c) Continuous neural fields, such as InstantNGP muller22, do not represent discontinuities, resulting in blur image when zoomed-in. (d) Denoising method, Field of Junctions (FoJ) verbin21, fails to recover clear discontinuities due to using constant-patch-based approximation. (e, f) Mumford-Shah functional based methods jointly approximate the target and detect discontinuities (see red edges in insets) wang22, similar to our method. However, both versions fail to achieve both goals because of limited function expressiveness. (g) Our accurate approximation and recovered discontinuities.
• Figure 3: (a) Our 1D feature basis functions fit discontinuous target function with continuous variables ($w, \mathbf{l}, \mathbf{r}$). (b) In a 2D vertex neighborhood, our 1D feature basis functions are defined along the radial direction. (c) The features have a constant piecewise slope per dimension, which is enhanced by an MLP.
• Figure 4: (a) Our feature (green) differs from belhe23's feature (cyan). (b) This allows us to easily discard almost-continuous edges.
• Figure 5: We initialize by triangulating Canny edges canny86 with TriWild hu19, then deforming and remeshing interleavedly. The deformation is posed as per-face constant color approximation.