Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking

TL;DR

Mamba4Net tackles the inefficiency of Transformer-based LLMs in networking by distilling networking-specific knowledge into a linear-time Mamba backbone. The framework introduces Domain Knowledge-Oriented Cross-Heterogeneous Distillation (DKO) and Cross-Heterogeneous Weight Reusing (CWR), guided by LoRA-based teacher adaptation, to enable stable cross-architecture knowledge transfer. Across viewport prediction, adaptive bitrate streaming, and cluster job scheduling, Mamba4Net achieves substantial efficiency gains (e.g., throughput 3.96× and model size 5.48% of prior LLM-based approaches) while preserving strong task performance. The work underscores the practicality of deploying compact, domain-tuned LLM-derived guidance on resource-constrained networks and provides open-source code to foster further research.

Abstract

Transformer-based large language models (LLMs) are increasingly being adopted in networking research to address domain-specific challenges. However, their quadratic time complexity and substantial model sizes often result in significant computational overhead and memory constraints, particularly in resource-constrained environments. Drawing inspiration from the efficiency and performance of the Deepseek-R1 model within the knowledge distillation paradigm, this paper introduces Mamba4Net, a novel cross-architecture distillation framework. Mamba4Net transfers networking-specific knowledge from transformer-based LLMs to student models built on the Mamba architecture, which features linear time complexity. This design substantially enhances computational efficiency compared to the quadratic complexity of transformer-based models, while the reduced model size further minimizes computational demands, improving overall performance and resource utilization. To evaluate its effectiveness, Mamba4Net was tested across three diverse networking tasks: viewport prediction, adaptive bitrate streaming, and cluster job scheduling. Compared to existing methods that do not leverage LLMs, Mamba4Net demonstrates superior task performance. Furthermore, relative to direct applications of transformer-based LLMs, it achieves significant efficiency gains, including a throughput 3.96 times higher and a storage footprint of only 5.48% of that required by previous LLM-based approaches. These results highlight Mamba4Net's potential to enable the cost-effective application of LLM-derived knowledge in networking contexts. The source code is openly available to support further research and development.

Figures (6)

• Figure 1: Framework of Mamba4Net including Multi-modal Encoders, Task-specific Networking Head, and Teacher–Student Model with Cross-Heterogeneous Weight Reusing (CWR).
• Figure 2: Structure diagram of Mamba and Transformer. In the diagram, there is a reuse relationship between weights of the same color in different architectures.
• Figure 3: Performance of Mamba4Net and its baselines in three various networking tasks.
• Figure 4: Model size and throughput of Mamba4Net compared against previous SOTA LLMs4Net approach NetLLM. In this evaluation, throughput only calculate the number of input token due to these special adaptions.
• Figure 5: Ablation study on pre-trained knowledge of LLMs. Mamba4Net-F denotes model with LLMs as teacher model, Mamba4Net-S denotes student model trained directly in networking tasks.