Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift

Yunpeng Tan, Qingyang Li, Mingxin Yang, Yannan Hu, Lei Zhang, Xinggong Zhang

TL;DR

MalMoE addresses encrypted traffic detection under temporal graph drift by employing a drift-aware Mixture-of-Experts framework. It builds two simple, drift-robust 1-hop-GNN-like experts on distinct node features (AVG and DEG) and uses a graph-informed gate to perform hard, explainable expert selection, aided by data augmentation and a two-stage training strategy. The approach delivers superior accuracy and F1 across public, synthetic, and real-world datasets while achieving real-time throughput on commodity hardware, demonstrating retraining-free, extensible drift resilience for ISP-scale deployments. These findings suggest that combining drift-robust representations with drift-aware routing offers a practical path for robust, scalable encrypted traffic detection.

Abstract

Encryption has been commonly used in network traffic to secure transmission, but it also brings challenges for malicious traffic detection, due to the invisibility of the packet payload. Graph-based methods are emerging as promising solutions by leveraging multi-host interactions to promote detection accuracy. But most of them face a critical problem: Graph Drift, where the flow statistics or topological information of a graph change over time. To overcome these drawbacks, we propose a graph-assisted encrypted traffic detection system, MalMoE, which applies Mixture of Experts (MoE) to select the best expert model for drift-aware classification. Particularly, we design 1-hop-GNN-like expert models that handle different graph drifts by analyzing graphs with different features. Then, the redesigned gate model conducts expert selection according to the actual drift. MalMoE is trained with a stable two-stage training strategy with data augmentation, which effectively guides the gate on how to perform routing. Experiments on open-source, synthetic, and real-world datasets show that MalMoE can perform precise and real-time detection.

MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift

TL;DR

MalMoE addresses encrypted traffic detection under temporal graph drift by employing a drift-aware Mixture-of-Experts framework. It builds two simple, drift-robust 1-hop-GNN-like experts on distinct node features (AVG and DEG) and uses a graph-informed gate to perform hard, explainable expert selection, aided by data augmentation and a two-stage training strategy. The approach delivers superior accuracy and F1 across public, synthetic, and real-world datasets while achieving real-time throughput on commodity hardware, demonstrating retraining-free, extensible drift resilience for ISP-scale deployments. These findings suggest that combining drift-robust representations with drift-aware routing offers a practical path for robust, scalable encrypted traffic detection.

Abstract

Encryption has been commonly used in network traffic to secure transmission, but it also brings challenges for malicious traffic detection, due to the invisibility of the packet payload. Graph-based methods are emerging as promising solutions by leveraging multi-host interactions to promote detection accuracy. But most of them face a critical problem: Graph Drift, where the flow statistics or topological information of a graph change over time. To overcome these drawbacks, we propose a graph-assisted encrypted traffic detection system, MalMoE, which applies Mixture of Experts (MoE) to select the best expert model for drift-aware classification. Particularly, we design 1-hop-GNN-like expert models that handle different graph drifts by analyzing graphs with different features. Then, the redesigned gate model conducts expert selection according to the actual drift. MalMoE is trained with a stable two-stage training strategy with data augmentation, which effectively guides the gate on how to perform routing. Experiments on open-source, synthetic, and real-world datasets show that MalMoE can perform precise and real-time detection.
Paper Structure (34 sections, 14 equations, 9 figures, 4 tables)

This paper contains 34 sections, 14 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Challenges of Graph-Aided Encrypted Traffic Detection
  4. Threat Model and Design Goals
  5. Insight for using MoE
  6. Method
  7. Overview
  8. Graph Construction and Expert Models
  9. Graph Construction
  10. Expert Models
  11. Gate Model
  12. Gate Model Input
  13. Gate Model Output
  14. Training of MalMoE
  15. Data Augmentation
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Drifts observed in the "flow number" and "bytes per packet" of a well-known backbone network operator.
  • Figure 2: Threat model of MalMoE's flow-level detection.
  • Figure 3: Accuracy and F1 score measured on test graphs with different graph drift, using different node features.
  • Figure 4: The overview of MalMoE.
  • Figure 5: The input and architecture of the expert models.
  • ...and 4 more figures