UTrice: Unifying Primitives in Differentiable Ray Tracing and Rasterization via Triangles for Particle-Based 3D Scenes

TL;DR

UTrice introduces a triangle-based differentiable ray tracing framework for particle-based 3D scenes, addressing proxy-geometry overhead by treating triangles as unified primitives for both ray tracing and rasterization. The method initializes triangles from SfM data, builds an OptiX BVH directly from triangles, and optimizes triangle geometry and appearance through a differentiable window function and a multi-term loss, achieving higher rendering quality than prior Gaussian-based approaches with near real-time performance. Key contributions include a differentiable triangle formulation, GPU-accelerated ray tracing without proxy geometries, and a training pipeline that unifies rasterization-based optimization with ray-traced rendering. Limitations include a larger primitive count and slower training due to unoptimized implementation, with future work targeting primitive reduction, connectivity preservation, and robustness to extreme triangles.

Abstract

Ray tracing 3D Gaussian particles enables realistic effects such as depth of field, refractions, and flexible camera modeling for novel-view synthesis. However, existing methods trace Gaussians through proxy geometry, which requires constructing complex intermediate meshes and performing costly intersection tests. This limitation arises because Gaussian-based particles are not well suited as unified primitives for both ray tracing and rasterization. In this work, we propose a differentiable triangle-based ray tracing pipeline that directly treats triangles as rendering primitives without relying on any proxy geometry. Our results show that the proposed method achieves significantly higher rendering quality than existing ray tracing approaches while maintaining real-time rendering performance. Moreover, our pipeline can directly render triangles optimized by the rasterization-based method Triangle Splatting, thus unifying the primitives used in novel-view synthesis.

Figures (10)

• Figure 1: Rendering results of our method. By directly using triangles as primitives, UTrice achieves higher rendering quality than 3DGRT 3dgrt while maintaining real-time performance (left and middle), and can produce realistic ray tracing effects such as depth of field (right). It also renders triangles optimized by 3DTS 3dts without additional processing, unifying the primitives used in rasterization and ray tracing based optimization.
• Figure 2: Overview of the pipeline. In the leftmost part, the triangles are initialized from the SfM point cloud, where each is generated by sampling three vertices within a unit sphere. They are then passed to OptiX for BVH construction and ray tracing, where each ray is visualized using its corresponding ground-truth pixel color. The triangles are iteratively optimized until the desired rendering quality is achieved and can subsequently be rendered using either a rasterizer or a ray tracer.
• Figure 3: Compatibility of our ray tracer. The response of the window function $I$ with the same smoothness factor $\sigma$ remains invariant under linear transformations. Therefore, our tracer can render triangles optimized by rasterization without any additional handling.
• Figure 4: Effect of $\sigma$ on the Triangle window function. Small $\sigma$ (e.g., $0.01$) makes the triangle nearly fully opaque. Larger $\sigma$ increases the sensitivity of the window function to interior-point variations, enabling gradient-based optimization of triangle geometry.
• Figure 5: Gradient propagation on triangle vertices. Gradients from the ray direction ($\partial r / \partial n$) propagate through the window function to the edge normals ($\partial n / \partial v$), updating vertex positions. This process results in triangle rotation and scaling toward the ground-truth shape. The detailed gradient derivations are provided in the supplementary material.