Low-Rank Adaptation of Neural Fields

TL;DR

The paper introduces Low-Rank Adaptation (LoRA) for neural fields, framing edits as compact, rank-constrained additive updates to base neural field weights. By representing changes with low-rank adapters, the method achieves substantial parameter savings (roughly 7–8x fewer parameters) while preserving high fidelity across image variations, geometric deformations, video encoding, and energy-based editing. It demonstrates predictable performance as a function of LoRA rank and variation magnitude, supports sequential edits for videos, and outperforms post-hoc low-rank baselines like SVD. The approach offers a lightweight, storage-efficient pathway for instance-specific neural field editing with broad practical implications for graphics, vision, and dynamic data processing.

Abstract

Processing visual data often involves small adjustments or sequences of changes, e.g., image filtering, surface smoothing, and animation. While established graphics techniques like normal mapping and video compression exploit redundancy to encode such small changes efficiently, the problem of encoding small changes to neural fields -- neural network parameterizations of visual or physical functions -- has received less attention. We propose a parameter-efficient strategy for updating neural fields using low-rank adaptations (LoRA). LoRA, a method from the parameter-efficient fine-tuning LLM community, encodes small updates to pre-trained models with minimal computational overhead. We adapt LoRA for instance-specific neural fields, avoiding the need for large pre-trained models and yielding lightweight updates. We validate our approach with experiments in image filtering, geometry editing, video compression, and energy-based editing, demonstrating its effectiveness and versatility for representing neural field updates.

Figures (14)

• Figure 1: We encode diverse types of variations---including (from left to right) surface deformations, image changes, energy-based denoising, videos, and physical simulation---as compact low-rank updates to pre-trained neural fields. Despite operating with $7-8\times$ fewer parameters than for full fine-tuning, our approach achieves visually faithful reconstructions across a range of tasks.
• Figure 2: Low-rank weight approximation error across layers of three fine-tuned image neural fields. A minor edit is performed to the target image of each neural field, and each neural field is fine-tuned to overfit to its edited image. The vertical axis measures the (normalized) approximation error of a rank-constrained factorization of the weight difference, accounting for the input distribution of the layer. A low rank approximation is sufficient to recover the fully fine-tuned weight with minimal error.
• Figure 3: Comparison between full fine-tuning (FT) and our LoRA-based approach for adapting a neural field to an image variation. The original image $\mathcal{D}$ is edited to produce $\mathcal{D}'$ by adding red marker strokes. LoRA reconstructs the edited image with high fidelity achieving a PSNR of 41.31 vs. 42.12 for FT while using $\approx$7.4$\times$ fewer parameters (46k vs. 340k). The inset shows the difference image between LoRA and FT outputs.
• Figure 4: Comparison of LoRA-based SDF reconstruction quality (IoU) using increasing rank $r$ against full fine-tuning (FT) for encoding surface deformations. The number of fine-tuning parameters as a percentage of the base model size is shown in the top row.
• Figure 5: Image (left) and SDF (right) reconstruction quality of LoRA vs. full-finetuning for image variations and surface deformations, respectively. Dotted line denotes full fine-tuning. See Figure \ref{['fig:main image result table']} and Figure \ref{['fig:sdf_ablation_reconstructions']} for the corresponding reconstructions.

Theorems & Definitions (1)

• Remark