Chat Modeling: Natural Language-based Procedural Modeling of Biological Structures without Training

TL;DR

The paper tackles the difficulty of using complex 3D modeling tools for biological structures by introducing Chat Modeling, a training-free framework that translates natural language into procedural modeling actions via a Modeling Translator. It presents a novel JSON-based code format, an LLM-driven code generator and interpreter, and an interactive user-refinement mechanism to iteratively improve outputs. The approach enables automatic and step-by-step modeling modes within the MesoCraft platform and is evaluated through expert feedback, showing high usability and potential for workflow integration in biosciences. The work suggests significant implications for making procedural, geometry-aware modeling more accessible to structural biologists and outlines directions for broader automation and user-driven refinement.

Abstract

3D modeling of biological structures is an inherently complex process, necessitating both biological and geometric understanding. Additionally, the complexity of user interfaces of 3D modeling tools and the associated steep learning curve further exacerbate the difficulty of authoring a 3D model. In this paper, we introduce a novel framework to address the challenge of using 3D modeling software by converting users' textual inputs into modeling actions within an interactive procedural modeling system. The framework incorporates a code generator of a novel code format and a corresponding code interpreter. The major technical innovation includes the user-refinement mechanism that captures the degree of user dissatisfaction with the modeling outcome, offers an interactive revision, and leverages this feedback for future improved 3D modeling. This entire framework is powered by large language models and eliminates the need for a traditional training process. We develop a prototype tool named Chat Modeling, offering both automatic and step-by-step 3D modeling approaches. Our evaluation of the framework with structural biologists highlights the potential of our approach being utilized in their scientific workflows. All supplemental materials are available at https://osf.io/x4qb7/.

Figures (8)

• Figure 1: Chat Modeling is a procedural modeling system that takes users' textual input and completes corresponding modeling operations. It could either complete the general modeling workflow (left), modify visual representations (top right), or self-correct the output by the user refinement mechanism (bottom right).
• Figure 2: Examples of rule types: for each rule, the left image shows rule creation and the right image shows the outcome after rule application.
• Figure 3: The framework starts with user text input, processed by the Modeling Translator, which creates prompts for an LLM and interprets its code into modeling actions. MesoCraft then models biological structures from these actions. Users visually inspect results and give feedback to improve modeling.
• Figure 4: The Modeling Translator consists of the code generator and the code interpreter. The code generator creates validated codes, while the code interpreter converts these codes into procedural modeling actions.
• Figure 5: User-refined few-shot prompting setting. The prompt includes a task description, initial examples, and user feedback examples.