Spline refinement with differentiable rendering

TL;DR

The paper tackles the challenge of refining slender, overlapping structures in microscopy by introducing a training-free, unsupervised differentiable rendering framework that jointly optimizes spline geometry, background, and texture to reconstruct the input image. This approach yields sub-pixel spline accuracy and robustness to distribution shifts, reducing the sim-to-real gap without requiring labeled data and acting as a drop-in replacement for traditional active contour refinement. Key contributions include a differentiable rendering pipeline for spline refinement, a three-phase optimization strategy, and demonstrations on C. elegans that show improvements over both coordinate-based and pixel-based methods, especially in overlapping and coiled morphologies. The work also presents a path toward assisted differentiable labeling and potential extensions to probabilistic priors and temporal 2D-splines, enabling broader applicability in quantitative microscopy and labeling workflows.

Abstract

Detecting slender, overlapping structures remains a challenge in computational microscopy. While recent coordinate-based approaches improve detection, they often produce less accurate splines than pixel-based methods. We introduce a training-free differentiable rendering approach to spline refinement, achieving both high reliability and sub-pixel accuracy. Our method improves spline quality, enhances robustness to distribution shifts, and shrinks the gap between synthetic and real-world data. Being fully unsupervised, the method is a drop-in replacement for the popular active contour model for spline refinement. Evaluated on C. elegans nematodes, a popular model organism for drug discovery and biomedical research, we demonstrate that our approach combines the strengths of both coordinate- and pixel-based methods.

Figures (4)

• Figure 1: Overview of the spline refinement method using differentiable rendering. The process starts with initial spline guesses, potentially from deep learning models. Through differentiable rasterization (orange arrows), an image is rendered from spline control points and, background and texture adjustments. The loss $\mathcal{L}$ is computed by comparing the rendered and input images, and gradients (blue arrows) update trainable parameters (bottom). At convergence, the refined spline (bottom right) accurately reconstructs the input shape (top right), as well as provides auxiliary data e.g. spline thickness.
• Figure 2: Robustness analysis of the proposed method on SOSB (64 frames). Average DTW after transformations: rotation (left), translation (middle), and scaling (right). The initial guess (yellow) is refined using active contour (red) or our method (blue). Shaded areas indicate worsened performance. Main plots show the top 50% performance, with insets displaying percentile variations. Some reconstruction examples are shown below.
• Figure 3: Examples of refinement (blue) of predictions (yellow) from (A) DeepTangleCrawlweheliye2024improved and from (B) de(ep)tanglealonsoFastDetectionSlender2023 on low-density C. elegans crawling experiments and high density swimming experiments, respectively. Example failure case is framed red.
• Figure 4: (A) Refining low-information labels (straight lines). Example framed in red shows a failure case. (B) Extracted scale parameter as a function of arc length, average in red.