APT-ClaritySet: A Large-Scale, High-Fidelity Labeled Dataset for APT Malware with Alias Normalization and Graph-Based Deduplication

TL;DR

The paper tackles the scarcity and inconsistency of large-scale APT malware benchmarks by introducing APT-ClaritySet, a curated pipeline combining alias normalization, graph-based deduplication, and function-level reuse analysis. It delivers three components: Full (34,363 samples, 305 groups), Unique (25,923 deduplicated samples, 303 groups), and FuncReuse (324,538 function-reuse clusters from over 9k samples), underpinned by a high-precision deduplication framework that uses CFG/FCG features and a hybrid structure-semantic similarity metric. A rigorous alias-mapping system achieves substantial label consistency (approx. 11.22% alias unification; 96.43% attribution accuracy on sampled labels) and enables scalable, reproducible attribution studies. The work further enables deep code-provenance insights through over 4.3 million analyzed functions forming the FRC resource, supporting analyses of inter-/intra-group sharing and tooling lineage, and it releases these assets with security and ethical safeguards for research use. Overall, APT-ClaritySet provides a high-quality, reproducible foundation for measuring APT patterns, evolution, and attribution, with broad implications for threat intel, defense, and future research.

Abstract

Large-scale, standardized datasets for Advanced Persistent Threat (APT) research are scarce, and inconsistent actor aliases and redundant samples hinder reproducibility. This paper presents APT-ClaritySet and its construction pipeline that normalizes threat actor aliases (reconciling approximately 11.22\% of inconsistent names) and applies graph-feature deduplication -- reducing the subset of statically analyzable executables by 47.55\% while retaining behaviorally distinct variants. APT-ClaritySet comprises: (i) APT-ClaritySet-Full, the complete pre-deduplication collection with 34{,}363 malware samples attributed to 305 APT groups (2006 - early 2025); (ii) APT-ClaritySet-Unique, the deduplicated release with 25{,}923 unique samples spanning 303 groups and standardized attributions; and (iii) APT-ClaritySet-FuncReuse, a function-level resource that includes 324{,}538 function-reuse clusters (FRCs) enabling measurement of inter-/intra-group sharing, evolution, and tooling lineage. By releasing these components and detailing the alias normalization and scalable deduplication pipeline, this work provides a high-fidelity, reproducible foundation for quantitative studies of APT patterns, evolution, and attribution.

Figures (7)

• Figure 1: Overview of the APT-ClaritySet Construction Methodology
• Figure 2: Overview of the Function-Level Clustering
• Figure 3: Workflow of the APT Alias Normalization and Mapping System
• Figure 4: Overall Workflow of the Graph-Feature-Based Binary Sample Deduplication Method
• Figure 5: Temporal Distribution of Major APT Groups by Sample Count