From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration

TL;DR

VisPainter addresses the challenge of producing editable, high-information-density scientific diagrams by bridging raster generative approaches and code-based vector tools with an interactive multi-agent system. It employs a Manager–Designer–Toolbox architecture operating via the Model Context Protocol to control a standard vector editor and generate element-level, editable diagrams, accompanied by VisBench, a streaming seven-dimensional benchmark for high-density schematics. The paper defines seven metrics spanning content fidelity, layout, readability, and interaction cost, and introduces a Dynamic Quality Score (DQS) that combines output quality and editing effort via $\mathrm{DQS}(s,n) = s(1-(1-s)\mathrm{sat}(n)) + r s (1-\mathrm{sat}(n))$, with $\mathrm{sat}(n) = \frac{n}{n+K}$. Through extensive ablations, it demonstrates the importance of role separation, a moderate step granularity, and description quality on performance, providing fair rankings across nine vision-language backbones. VisPainter and VisBench thereby offer a practical framework and evaluation protocol to enable efficient, observable, and editable scientific diagram creation.

Abstract

Scientific illustrations demand both high information density and post-editability. However, current generative models have two major limitations: Frist, image generation models output rasterized images lacking semantic structure, making it impossible to access, edit, or rearrange independent visual components in the images. Second, code-based generation methods (TikZ or SVG), although providing element-level control, force users into the cumbersome cycle of "writing-compiling-reviewing" and lack the intuitiveness of manipulation. Neither of these two approaches can well meet the needs for efficiency, intuitiveness, and iterative modification in scientific creation. To bridge this gap, we introduce VisPainter, a multi-agent framework for scientific illustration built upon the model context protocol. VisPainter orchestrates three specialized modules-a Manager, a Designer, and a Toolbox-to collaboratively produce diagrams compatible with standard vector graphics software. This modular, role-based design allows each element to be explicitly represented and manipulated, enabling true element-level control and any element can be added and modified later. To systematically evaluate the quality of scientific illustrations, we introduce VisBench, a benchmark with seven-dimensional evaluation metrics. It assesses high-information-density scientific illustrations from four aspects: content, layout, visual perception, and interaction cost. To this end, we conducted extensive ablation experiments to verify the rationality of our architecture and the reliability of our evaluation methods. Finally, we evaluated various vision-language models, presenting fair and credible model rankings along with detailed comparisons of their respective capabilities. Additionally, we isolated and quantified the impacts of role division, step control,and description on the quality of illustrations.

Figures (34)

• Figure 1: VisPainter framework diagram and workflow diagram.The overall workflow proceeds as follows: (1) the Manager parses the user request into a structured task and locates the relevant Visio functions; (2) the Designer generates an initial draft layout; (3) the Manager invokes Toolbox operations to render the draft and captures a screenshot; (4) the Designer iteratively updates the layout based on feedback until convergence. The final output includes both a bitmap preview for quick inspection and a vector source file that can be further modified within Visio or other editors.
• Figure 2: Examples of scientific plotting outputs from different models.
• Figure 3: Comparison of 9 model capabilities: T2I test scenario (a); TI2I test scenario (b). Comparison of model capabilities before and after role configuration ablation experiments: T2I test scenario (c); TI2I test scenario (d). "mer" stands for role integration, and "ori" stands for baseline setting.
• Figure 4: Visualization charts of the ablation experiments for description quality: T2I test scenario (a); TI2I test scenario (b). The radar chart represents the capability comparison between different models in this test, and the heatmap represents the changes in model capabilities compared to the standard description quality.
• Figure 5: Experimental results of difficulty-balanced sampling. The upper part shows the T2I test scenario, and the lower part shows the TI2I test scenario. From left to right are the data distribution, stability curve, and data coverage heatmap.