INST-Sculpt: Interactive Stroke-based Neural SDF Sculpting

TL;DR

This work addresses the challenge of editing neural implicit surfaces by introducing INST-Sculpt, a real-time stroke-based sculpting framework for pretrained neural $SDF$s. It combines a tubular sampling strategy with a flexible brush model and a dedicated $u$-$v$-$n$ coordinate frame to apply smooth, localized deformations to the zero-level set, without reparameterizing the network. A SIREN-based MLP models the neural $SDF$, trained with a multi-term loss that enforces regression accuracy, the eikonal equation, normals alignment, and boundary regularization. Empirical results show up to 16x speedups over point-based edits, improved fidelity over baselines, and interactive performance across diverse shapes, enabling expressive, stroke-driven editing of neural implicit geometry with potential impact on graphics, CAD, and AR workflows.

Abstract

Recent advances in implicit neural representations have made them a popular choice for modeling 3D geometry, achieving impressive results in tasks such as shape representation, reconstruction, and learning priors. However, directly editing these representations poses challenges due to the complex relationship between model weights and surface regions they influence. Among such editing tools, sculpting, which allows users to interactively carve or extrude the surface, is a valuable editing operation to the graphics and modeling community. While traditional mesh-based tools like ZBrush facilitate fast and intuitive edits, a comparable toolkit for sculpting neural SDFs is currently lacking. We introduce a framework that enables interactive surface sculpting edits directly on neural implicit representations. Unlike previous works limited to spot edits, our approach allows users to perform stroke-based modifications on the fly, ensuring intuitive shape manipulation without switching representations. By employing tubular neighborhoods to sample strokes and custom brush profiles, we achieve smooth deformations along user-defined curves, providing precise control over the sculpting process. Our method demonstrates that intricate and versatile edits can be made while preserving the smooth nature of implicit representations.

Figures (7)

• Figure 1: Overview of our interactive neural sculpting pipeline. 1: User defines a custom brush profile by selecting control points in a $[-1,1]$ window, interpolated with a cubic spline. 2: A brush stroke is drawn on the neural SDF-rendered shape in an interactive viewer with darker regions indicating a higher offset distance. 3: A moving $u$-$v$-$n$ coordinate frame and tubular sampling region are established around the stroke path. 4: Samples are used to query the SDF’s MLP for fine-tuning. 5: Resulting deformations, in this case, realistic collarbones, are shown from multiple angles (front, cutaway, side and isometric views).
• Figure 2: Examples of natural artistic edits using our method. Each column shows a sample shape from the dataset, with the original SDF on the first row and the edited version below. (a) Ring with a central gem and tapering carvings. (b) Halloween-themed pumpkin carving. (c) Frog morphed into a lizard with scales using a sine-modulated stroke. (d) Single-stroke cursive 'cvpr' inscription on a sphere. (e) Bunny transformed with robotic features on its face, feet, and ears. (f) Victorian hairstyle added to a previously bald bust.
• Figure 3: Examples of edits using different brush profiles and modulation functions. Above each shape, the left plot shows the brush profile, and the right plot shows the modulation.
• Figure 4: Effect of radius and intensity variations for the same brush stroke.
• Figure 5: Illustration of the coordinate frame and sampling strategy. Left: A 3D view of the user-defined stroke with control points and the $u$-$v$-$n$ frame on the surface. Right: A top-down view of the $u$-$v$ plane showing tubular sampling within a radius around the stroke.