LLMs, You Can Evaluate It! Design of Multi-perspective Report Evaluation for Security Operation Centers

TL;DR

The paper tackles the challenge of evaluating security operation center (SOC) analysis reports produced from security alerts, arguing that expert-aligned evaluation is essential to ensure report quality and actionable insights.It introduces MESSALA, a novel framework that combines an Analyst-wise checklist (constructed from literature and practitioner interviews) with a Granularization Guideline and a Multi-perspective Evaluation LLM to produce quantitative scores and qualitative feedback aligned with veteran analysts.Through extensive experiments on real-world and pseudo SOC reports, MESSALA achieves higher alignment with human expert judgments than baseline LLM-based methods and demonstrates the ability to provide actionable feedback, including defect detection in reports.Qualitative evaluations further show that MESSALA delivers more specific, understandable, and practitioner-relevant feedback, supporting both novice and veteran analysts, though some limitations in accuracy and context prioritization remain.Overall, the framework promises practical impact for SOC operations by reducing evaluation workload, improving report quality, and enabling more reliable, expert-informed feedback to report authors.

Abstract

Security operation centers (SOCs) often produce analysis reports on security incidents, and large language models (LLMs) will likely be used for this task in the near future. We postulate that a better understanding of how veteran analysts evaluate reports, including their feedback, can help produce analysis reports in SOCs. In this paper, we aim to leverage LLMs for analysis reports. To this end, we first construct a Analyst-wise checklist to reflect SOC practitioners' opinions for analysis report evaluation through literature review and user study with SOC practitioners. Next, we design a novel LLM-based conceptual framework, named MESSALA, by further introducing two new techniques, granularization guideline and multi-perspective evaluation. MESSALA can maximize report evaluation and provide feedback on veteran SOC practitioners' perceptions. When we conduct extensive experiments with MESSALA, the evaluation results by MESSALA are the closest to those of veteran SOC practitioners compared with the existing LLM-based methods. We then show two key insights. We also conduct qualitative analysis with MESSALA, and then identify that MESSALA can provide actionable items that are necessary for improving analysis reports.

Figures (10)

• Figure 1: Overview of MESSALA. The top module runs two parallel evaluators: a High-level LLM and an In-depth LLM guided by the Granularization Guideline; their outputs are fused by the Multi-perspective Evaluation LLM.
• Figure 2: Granularization guideline. The granularization guideline can break down items in the checklist into specific evaluation steps for each category.
• Figure 3: Violin plots illustrating the distribution of method-wise evaluation scores produced by each model, compared with human gold standard evaluations.
• Figure 4: Comparison of Evaluation Comments: Opaque Decision Rationale category
• Figure 5: Comparison of Evaluation Comments: Unverifiable or One-Sided Analysis category