Diffusion Models Meet Network Management: Improving Traffic Matrix Analysis with Diffusion-based Approach

TL;DR

The paper tackles the problem of estimating traffic matrices (TM) from partial measurements in large networks, where direct full measurement is prohibitively costly. It introduces Diffusion-TM, a diffusion-based, plug-and-play framework that learns an unconditional TM distribution and uses conditional guidance from measurements via Tweedie-inspired updates to generate measurement-consistent TM estimates, supported by EM and replacement-based mechanisms for tomography and completion. A two-stage training pipeline with an autoencoder pre-processing step enables diffusion modeling under heavy missing-data regimes, while a Transformer backbone handles high-dimensional TM sequences. Across Abilene and GÉANT datasets, Diffusion-TM demonstrates superior distribution fidelity and TM reconstruction accuracy (TM synthesis, completion, tomography) with competitive compute times, offering a practical, distribution-aware alternative to traditional low-rank and routing-based TM methods; however, the iterative denoising process remains computationally intensive.

Abstract

Due to network operation and maintenance relying heavily on network traffic monitoring, traffic matrix analysis has been one of the most crucial issues for network management related tasks. However, it is challenging to reliably obtain the precise measurement in computer networks because of the high measurement cost, and the unavoidable transmission loss. Although some methods proposed in recent years allowed estimating network traffic from partial flow-level or link-level measurements, they often perform poorly for traffic matrix estimation nowadays. Despite strong assumptions like low-rank structure and the prior distribution, existing techniques are usually task-specific and tend to be significantly worse as modern network communication is extremely complicated and dynamic. To address the dilemma, this paper proposed a diffusion-based traffic matrix analysis framework named Diffusion-TM, which leverages problem-agnostic diffusion to notably elevate the estimation performance in both traffic distribution and accuracy. The novel framework not only takes advantage of the powerful generative ability of diffusion models to produce realistic network traffic, but also leverages the denoising process to unbiasedly estimate all end-to-end traffic in a plug-and-play manner under theoretical guarantee. Moreover, taking into account that compiling an intact traffic dataset is usually infeasible, we also propose a two-stage training scheme to make our framework be insensitive to missing values in the dataset. With extensive experiments with real-world datasets, we illustrate the effectiveness of Diffusion-TM on several tasks. Moreover, the results also demonstrate that our method can obtain promising results even with $5\%$ known values left in the datasets.

Figures (12)

• Figure 1: Illustration of studied problems in this paper. We seek an estimated TM $\boldsymbol{X}$ that satisfies the conditions imposed by the set of measurements $\boldsymbol{R}$ or $\boldsymbol{Y}$. However, the considered problem is highly underdetermined .
• Figure 2: Geometrical visualization of diffusion models. The central area represents the original data manifold which has been proved to be encircled by manifolds of noisy data $p_t\left( \boldsymbol{x_t}\right)$r31. The encoding (forward) process depicted by orange arrows gradually converts original data distribution $p_0\left( \boldsymbol{x_0}\right)$, into a simple isotropic Gaussian ${\cal N}\left(0,\rm{I}\right)$. While the decoding (reverse) process depicted by green arrows can be considered as transitions from $p_t\left( \boldsymbol{x_t}\right)$ to $p_{t-1}\left( \boldsymbol{x_{t-1}}\right)$ through a Markov Chain.
• Figure 3: Illustration of our diffusion-based approach for solving TM estimation problems. The reverse inference process (from right to left) iteratively denoises the target traffic matrix $\boldsymbol{x_0}$ conditioned on the measurement $\boldsymbol{y}$. Concretely, following the prediction of the estimated $\boldsymbol{\hat{x}_0}$ by an unconditional diffusion model, the measurement $\boldsymbol{y}$ is incorporated by solving a proximal subproblem depicted by red arrows in the VP-SDE.
• Figure 4: Guiding generation process toward target solutions. Each curve represents a manifold ${\cal M}_i$ of (noisy) TM data. The proposed correction step (red arrow) alleviates reverse diffusion step (green arrow) leaving the solution space of inverse problems.
• Figure 5: (Top) Architecture of Pre-processing Module. The training of diffusion models starts with a AutoEncoder-based pre-processing module, which generates coarse-grained estimations of missing values in training set. (Bottom) Overall model structure of Diffusion-TM. The underlying Transformer is fed a TM sequence $x_t$ in a diffusion step $t$, as well as $t$ itself, then the diffusion model predicts the clean sample ${\hat{x}}_0$.