PlantDreamer: Achieving Realistic 3D Plant Models with Diffusion-Guided Gaussian Splatting

TL;DR

PlantDreamer addresses the challenge of realistic 3D plant generation for phenotyping by introducing a diffusion-guided Gaussian Splatting workflow that initializes from L-System meshes or real point clouds. It integrates depth conditioning via ControlNet, species-specific LoRA texture transfer, and a Gaussian culling mechanism to preserve geometry while enhancing surface realism. The method demonstrates superior cross-view consistency and texture fidelity compared with state-of-the-art text-to-3D models, and it enables conversion of real plant scans into dense 3DGS representations. These capabilities advance 3D plant phenotyping by providing high-quality synthetic data and improved tooling for upgrading legacy point clouds.

Abstract

Recent years have seen substantial improvements in the ability to generate synthetic 3D objects using AI. However, generating complex 3D objects, such as plants, remains a considerable challenge. Current generative 3D models struggle with plant generation compared to general objects, limiting their usability in plant analysis tools, which require fine detail and accurate geometry. We introduce PlantDreamer, a novel approach to 3D synthetic plant generation, which can achieve greater levels of realism for complex plant geometry and textures than available text-to-3D models. To achieve this, our new generation pipeline leverages a depth ControlNet, fine-tuned Low-Rank Adaptation and an adaptable Gaussian culling algorithm, which directly improve textural realism and geometric integrity of generated 3D plant models. Additionally, PlantDreamer enables both purely synthetic plant generation, by leveraging L-System-generated meshes, and the enhancement of real-world plant point clouds by converting them into 3D Gaussian Splats. We evaluate our approach by comparing its outputs with state-of-the-art text-to-3D models, demonstrating that PlantDreamer outperforms existing methods in producing high-fidelity synthetic plants. Our results indicate that our approach not only advances synthetic plant generation, but also facilitates the upgrading of legacy point cloud datasets, making it a valuable tool for 3D phenotyping applications.

Figures (8)

• Figure 1: Rendered images of different synthetic plants generated using PlantDreamer. The images are categorised by plant species: bean, mint, and kale, respectively.
• Figure 2: Overview of the process for generating a 3D bean model using PlantDreamer. Our approach either accepts a L-System mesh or a point cloud from a real plant, and outputs a 3DGS model, which is shown for the synthetic and real plant on the right side of the diagram.
• Figure 3: Our real bean, kale and mint captured point clouds with the corresponding synthetic meshes.
• Figure 4: A comparison between the ground truth and 3D plant models generated by PlantDreamer and GaussianDreamer. Each model was initialised with the same real point cloud for the bean, kale and mint respectively.
• Figure 5: Visual comparison between rendered outputs for two selected synthetic bean, kale and mint plants for Latent-NeRF, Magic3D, Fantasia3D models, GaussianDreamer and PlantDreamer. Each model was initialised from their native methods.