Toward Automated and Trustworthy Scientific Analysis and Visualization with LLM-Generated Code
Apu Kumar Chakroborti, Yi Ding, Lipeng Wan
TL;DR
Problem: domain scientists face barriers to programmatic data analysis and visualization; LLMs offer potential but reliability is uncertain. Approach: construct a benchmark across hierarchical scientific datasets and evaluate three strategies—data-aware prompt disambiguation, retrieval-augmented prompt enhancement, and iterative error repair—to improve autonomous code generation. Findings: these techniques increase script executability and some output quality, but gaps remain in domain-specific correctness; improvements are dataset- and model-dependent. Significance: the work provides actionable methods and a reusable benchmark to advance trustworthy LLM-assisted scientific workflows.
Abstract
As modern science becomes increasingly data-intensive, the ability to analyze and visualize large-scale, complex datasets is critical to accelerating discovery. However, many domain scientists lack the programming expertise required to develop custom data analysis workflows, creating barriers to timely and effective insight. Large language models (LLMs) offer a promising solution by generating executable code from natural language descriptions. In this paper, we investigate the trustworthiness of open-source LLMs in autonomously producing Python scripts for scientific data analysis and visualization. We construct a benchmark suite of domain-inspired prompts that reflect real-world research tasks and systematically evaluate the executability and correctness of the generated code. Our findings show that, without human intervention, the reliability of LLM-generated code is limited, with frequent failures caused by ambiguous prompts and the models' insufficient understanding of domain-specific contexts. To address these challenges, we design and assess three complementary strategies: data-aware prompt disambiguation, retrieval-augmented prompt enhancement, and iterative error repair. While these methods significantly improve execution success rates and output quality, further refinement is needed. This work highlights both the promise and current limitations of LLM-driven automation in scientific workflows and introduces actionable techniques and a reusable benchmark for building more inclusive, accessible, and trustworthy AI-assisted research tools.