Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models

Ron F. Del Rosario

TL;DR

This work tackles the challenge of detecting temporal attack patterns in multi-agent AI workflows by delivering an open end-to-end framework for training trace-based security models. It combines multi-source data curation with synthetic OpenTelemetry trace generation and lean QLoRA fine-tuning on ARM64 hardware to achieve substantial gains on cybersecurity knowledge benchmarks (from 42.86% to 74.29% overall). However, practical trace analysis reveals high false positives due to data distribution bias, underscoring the need for architectural remedies such as balanced retraining or RAG-based context augmentation before production deployment. By releasing all datasets, code, and benchmarks on HuggingFace, the paper enables reproducibility and community-driven advancement in agentic security research. The results demonstrate both the promise of trace-based approaches for dynamic security and the caution required for real-world deployment where human oversight remains essential.

Abstract

We present an openly documented methodology for fine-tuning language models to detect temporal attack patterns in multi-agent AI workflows using OpenTelemetry trace analysis. We curate a dataset of 80,851 examples from 18 public cybersecurity sources and 35,026 synthetic OpenTelemetry traces. We apply iterative QLoRA fine-tuning on resource-constrained ARM64 hardware (NVIDIA DGX Spark) through three training iterations with strategic augmentation. Our custom benchmark accuracy improves from 42.86% to 74.29%, a statistically significant 31.4-point gain. Targeted examples addressing specific knowledge gaps outperform indiscriminate scaling. Key contributions include: (1) synthetic trace generation methodology for multi-agent coordination attacks and regulatory violations, (2) empirical evidence that training data composition fundamentally determines behavior, and (3) complete open release of datasets, training scripts, and evaluation benchmarks on HuggingFace. While practical deployment requires human oversight due to false positive rates, this work establishes the first reproducible framework enabling practitioners to build custom agentic security models adapted to their threat landscapes.

Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models

TL;DR

This work tackles the challenge of detecting temporal attack patterns in multi-agent AI workflows by delivering an open end-to-end framework for training trace-based security models. It combines multi-source data curation with synthetic OpenTelemetry trace generation and lean QLoRA fine-tuning on ARM64 hardware to achieve substantial gains on cybersecurity knowledge benchmarks (from 42.86% to 74.29% overall). However, practical trace analysis reveals high false positives due to data distribution bias, underscoring the need for architectural remedies such as balanced retraining or RAG-based context augmentation before production deployment. By releasing all datasets, code, and benchmarks on HuggingFace, the paper enables reproducibility and community-driven advancement in agentic security research. The results demonstrate both the promise of trace-based approaches for dynamic security and the caution required for real-world deployment where human oversight remains essential.

Abstract

We present an openly documented methodology for fine-tuning language models to detect temporal attack patterns in multi-agent AI workflows using OpenTelemetry trace analysis. We curate a dataset of 80,851 examples from 18 public cybersecurity sources and 35,026 synthetic OpenTelemetry traces. We apply iterative QLoRA fine-tuning on resource-constrained ARM64 hardware (NVIDIA DGX Spark) through three training iterations with strategic augmentation. Our custom benchmark accuracy improves from 42.86% to 74.29%, a statistically significant 31.4-point gain. Targeted examples addressing specific knowledge gaps outperform indiscriminate scaling. Key contributions include: (1) synthetic trace generation methodology for multi-agent coordination attacks and regulatory violations, (2) empirical evidence that training data composition fundamentally determines behavior, and (3) complete open release of datasets, training scripts, and evaluation benchmarks on HuggingFace. While practical deployment requires human oversight due to false positive rates, this work establishes the first reproducible framework enabling practitioners to build custom agentic security models adapted to their threat landscapes.
Paper Structure (40 sections, 3 figures)

This paper contains 40 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset Acquisition and Curation
  5. Model Architecture and Training
  6. Baseline Comparison
  7. Experimental Setup
  8. Results and Evaluation
  9. Training Metrics
  10. MMLU Computer Security Benchmark
  11. Custom Cybersecurity MCQA: Iterative Improvement Journey
  12. Practical Trace Analysis Validation
  13. Quantitative Results
  14. Ablation Study: Why Prompt Engineering Cannot Fix Training Bias
  15. Discussion
  16. ...and 25 more sections

Figures (3)

  • Figure 1: Training Loss Curves V2/V3/V4
  • Figure 2: V2→V3→V4 Accuracy Progression
  • Figure 3: Confusion Matrix - 60% TPR, 0% TNR, 66.7% FPR