ScribbleSense: Generative Scribble-Based Texture Editing with Intent Prediction

TL;DR

The paper tackles the challenge of ambiguous intent in scribble-based texture editing for 3D meshes. It introduces ScribbleSense, a pipeline that uses multimodal large language models to infer editing intent from multi-view scribbles and global context, then generates coherent local textures via global prompts and diffusion, guided by geometry-aware masking. A geometry-guided refinement step and a grid-based texture patch integration with inpainting ensure precise spatial control and seamless blending across viewpoints. Experiments on multiple 3D meshes demonstrate competitive quantitative performance and strong user preference, with the approach achieving around 16 seconds per edit on an A100 and showing robust generalization across MLLMs and editing scenarios.

Abstract

Interactive 3D model texture editing presents enhanced opportunities for creating 3D assets, with freehand drawing style offering the most intuitive experience. However, existing methods primarily support sketch-based interactions for outlining, while the utilization of coarse-grained scribble-based interaction remains limited. Furthermore, current methodologies often encounter challenges due to the abstract nature of scribble instructions, which can result in ambiguous editing intentions and unclear target semantic locations. To address these issues, we propose ScribbleSense, an editing method that combines multimodal large language models (MLLMs) and image generation models to effectively resolve these challenges. We leverage the visual capabilities of MLLMs to predict the editing intent behind the scribbles. Once the semantic intent of the scribble is discerned, we employ globally generated images to extract local texture details, thereby anchoring local semantics and alleviating ambiguities concerning the target semantic locations. Experimental results indicate that our method effectively leverages the strengths of MLLMs, achieving state-of-the-art interactive editing performance for scribble-based texture editing.

Figures (15)

• Figure 1: By freely scribbling with different colors on the surface of the 3D model, our model can reasonably predict the user's editing intentions and generate high-quality local textures that match the scribble colors.
• Figure 2: Overview of ScribbleSense. We leverage multimodal LLMs in our pipeline for semantic prediction, global prompt generation, and local texture selection for texture editing.
• Figure 3: For the same local predicted semantics, using different global prompts will generate different forms of the object.
• Figure 4: Examples of dialogues with the LLM at different stages of the pipeline.
• Figure 5: The rendering of textureless 3D meshes highlights the geometric structure of the model. To process the user's rough scribble, our method combines this geometric information with a segmentation model to iteratively refine the scribbled area mask.