Data Analysis and Performance Evaluation of Simulation Deduction Based on LLMs
Shansi Zhang, Min Li
TL;DR
The paper addresses the challenge of producing high-quality, well-formatted analysis reports from military simulation deduction data, which is poorly served by single-pass LLM outputs. It introduces a two-part framework: (i) task decomposition with specialized system and user prompts, and (ii) multi-round LLM interactions with external tools for plotting and metric computation, culminating in complete reports through template-based templates. The approach yields five report templates tailored to different data inputs, with extensive evaluations showing superior report quality and formatting over a baseline lacking task decomposition and tool use, especially as report complexity increases. The method demonstrates practical impact by enabling faster, more reliable, and adaptable analytical reporting in data-driven military environments.
Abstract
Data analysis and performance evaluation of simulation deduction plays a pivotal role in modern warfare, which enables military personnel to gain invaluable insights into the potential effectiveness of different strategies, tactics, and operational plans. Traditional manual analysis approach is time-consuming and limited by human errors. To enhance efficiency and accuracy, large language models (LLMs) with strong analytical and inferencing capabilities can be employed. However, high-quality analysis reports with well-structured formatting cannot be obtained through a single instruction input to the LLM. To tackle this issue, we propose a method that first decomposes the complex task into several sub-tasks and designs effective system prompts and user prompts for each sub-task. Multi-round interactions with the LLM incorporating self-check and reflection are then conducted to enable structured data extraction as well as multi-step analysis and evaluation. Furthermore, custom tools are defined and invoked to generate figures and compute metrics. We also design multiple report templates, each tailored to a specific application and input data type, ensuring their adaptability across a variety of scenarios. Extensive evaluation results demonstrate that the reports generated by our method exhibit higher quality, therefore obtaining higher scores than the baseline method.