UniMGS: Unifying Mesh and 3D Gaussian Splatting with Single-Pass Rasterization and Proxy-Based Deformation

TL;DR

UniMGS tackles the challenge of jointly rendering and deforming meshes and 3D Gaussian Splatting (3DGS) by introducing a single-pass, anti-aliased rasterization pipeline and a Gaussian-centric binding strategy to a proxy mesh. The framework unifies rendering of both representations via α-blending in one pass and binds Gaussians to mesh faces in a training-free, topology-robust manner, with deformation transferred through BBX-aware transformations. Experiments on diverse datasets show superior novel-view quality, robust deformation under imperfect proxies, and practical applicability to embodied AI and fluid simulation. The work advances hybrid representations by enabling accurate occlusion, transparency, and deformation without retraining, and it provides code to accelerate future research.

Abstract

Joint rendering and deformation of mesh and 3D Gaussian Splatting (3DGS) have significant value as both representa tions offer complementary advantages for graphics applica tions. However, due to differences in representation and ren dering pipelines, existing studies render meshes and 3DGS separately, making it difficult to accurately handle occlusions and transparency. Moreover, the deformed 3DGS still suffers from visual artifacts due to the sensitivity to the topology quality of the proxy mesh. These issues pose serious obsta cles to the joint use of 3DGS and meshes, making it diffi cult to adapt 3DGS to conventional mesh-oriented graphics pipelines. We propose UniMGS, the first unified framework for rasterizing mesh and 3DGS in a single-pass anti-aliased manner, with a novel binding strategy for 3DGS deformation based on proxy mesh. Our key insight is to blend the col ors of both triangle and Gaussian fragments by anti-aliased α-blending in a single pass, achieving visually coherent re sults with precise handling of occlusion and transparency. To improve the visual appearance of the deformed 3DGS, our Gaussian-centric binding strategy employs a proxy mesh and spatially associates Gaussians with the mesh faces, signifi cantly reducing rendering artifacts. With these two compo nents, UniMGS enables the visualization and manipulation of 3D objects represented by mesh or 3DGS within a unified framework, opening up new possibilities in embodied AI, vir tual reality, and gaming. We will release our source code to facilitate future research.

Figures (7)

• Figure 1: Differences between UniMGS and prior work in rasterization and deformation.
• Figure 2: Overview of UniMGS. Given objects composed of 3DGS and mesh, the unified rendering pipeline bridges Gaussians and triangles by $\alpha$-blending in a single-pass manner, thus accurately computing color and handling occlusion. To mitigate aliasing artifacts, we group depth-adjacent triangle fragments as a single entity, in which $\alpha$-blending with MSAA is performed. The framework further allows mesh-based deformation to be seamlessly extended to 3DGS with a proxy mesh. We associate a Gaussian with triangle faces by projecting the vertices of its BBX onto the mesh. During deformation, the motion of triangle faces is first transferred to the BBX and then propagated to the Gaussian.
• Figure 3: Illustration of single-pass rasterization with $\alpha$-blending. $\to$ means the update of transmittance while $\uparrow$ represents the transmittance used in $\alpha$-blending. (a): Directly blend triangles and Gaussians without anti-aliasing of triangles. (b): Consider all triangles overlapping the same pixel as a whole and perform MSAA in $\alpha$-blending. (c): Apply (b) directly to (a) causes color overflow. (d): Modify (c) by treating only depth-adjacent triangles as an entity, where the difference is highlighted in red.
• Figure 4: Visual improvements brought by our rasterization pipeline. (a): Direct $\alpha$-blending; (b): Aliasing removed by combining MSAA with $\alpha$-blending; (c): Color overflow caused by incorrect transmittance; (d): Our full method without artifacts.
• Figure 5: Evaluation against existing hybrid rendering studies. (a): Compared to ray-based methods, our method shows promising runtime performance while maintaining comparable visual effects. (b): Compared to separate-pass rasterization work, our method guarantees the correct spatial relationship because the colors of Gaussians and triangles are blended in a single pass.