Decision-Aware Trust Signal Alignment for SOC Alert Triage

TL;DR

The paper tackles the problem of miscalibrated and decision-inconsistent confidence signals in SOC alert triage, where asymmetric costs penalize missed attacks more than false alarms. It introduces a model-agnostic, decision-aware trust signal alignment framework that uses post-hoc calibration and explicit uncertainty cues to map detector outputs to cost-sensitive escalation decisions without modifying the underlying detection models, formalized through $t^{*}=\frac{C_{FP}}{C_{FP}+C_{FN}}$ and calibrated probability $p_{cal}$. The approach is validated on Logistic Regression and Random Forest with UNSW-NB15, showing that aligned trust interfaces significantly reduce false negatives and cost-weighted loss across models, while misaligned interfaces worsen performance. A human-in-the-loop evaluation plan is proposed to empirically assess analyst decision-making under aligned versus misaligned trust, underscoring the practical impact of interface design on SOC effectiveness and suggesting model-agnostic paths to safer human-AI collaboration in security contexts.

Abstract

Detection systems that utilize machine learning are progressively implemented at Security Operations Centers (SOCs) to help an analyst to filter through high volumes of security alerts. Practically, such systems tend to reveal probabilistic results or confidence scores which are ill-calibrated and hard to read when under pressure. Qualitative and survey based studies of SOC practice done before reveal that poor alert quality and alert overload greatly augment the burden on the analyst, especially when tool outputs are not coherent with decision requirements, or signal noise. One of the most significant limitations is that model confidence is usually shown without expressing that there are asymmetric costs in decision making where false alarms are much less harmful than missed attacks. The present paper presents a decision-sensitive trust signal correspondence scheme of SOC alert triage. The framework combines confidence that has been calibrated, lightweight uncertainty cues, and cost-sensitive decision thresholds into coherent decision-support layer, instead of making changes to detection models. To enhance probabilistic consistency, the calibration is done using the known post-hoc methods and the uncertainty cues give conservative protection in situations where model certainty is low. To measure the model-independent performance of the suggested model, we apply the Logistic Regression and the Random Forest classifiers to the UNSW-NB15 intrusion detection benchmark. According to simulation findings, false negatives are greatly amplified by the presence of misaligned displays of confidence, whereas cost weighted loss decreases by orders of magnitude between models with decision aligned trust signals. Lastly, we describe a human-in-the-loop study plan that would allow empirically assessing the decision-making of the analysts with aligned and misaligned trust interfaces.

Figures (4)

• Figure 1: Overview of the proposed decision-aware trust signal alignment framework for SOC alert triage.
• Figure 2: Reliability diagrams comparing raw and calibrated probability estimates. The dashed diagonal indicates perfect calibration.
• Figure 3: Cost-weighted loss as a function of the escalation threshold using calibrated probabilities. The dashed vertical line indicates the decision-aware threshold $t^* = 0.0909$ derived from asymmetric costs ($C_{FN}=10$, $C_{FP}=1$).
• Figure :