Automated Network Protocol Testing with LLM Agents

TL;DR

The paper tackles the challenge of labor-intensive network protocol testing by introducing NeTestLLM, a multi-agent LLM framework for end-to-end automation. It combines hierarchical RFC understanding, iterative test-case generation with verification, domain-specific artifact generation, and runtime feedback analysis to produce executable tester scripts and DUT configurations with minimal human input. Empirical results show substantial improvements in coverage (e.g., 41 FRRouting bugs exposed versus 11 in national standards) and efficiency (artifact generation speeding up manual methods by about 8.65x), validated through production deployment and expert evaluations. The work demonstrates a practical, scalable solution for automated testing of heterogeneous network protocols, with strong potential to adapt to evolving standards and device ecosystems.

Abstract

Network protocol testing is fundamental for modern network infrastructure. However, traditional network protocol testing methods are labor-intensive and error-prone, requiring manual interpretation of specifications, test case design, and translation into executable artifacts, typically demanding one person-day of effort per test case. Existing model-based approaches provide partial automation but still involve substantial manual modeling and expert intervention, leading to high costs and limited adaptability to diverse and evolving protocols. In this paper, we propose a first-of-its-kind system called NeTestLLM that takes advantage of multi-agent Large Language Models (LLMs) for end-to-end automated network protocol testing. NeTestLLM employs hierarchical protocol understanding to capture complex specifications, iterative test case generation to improve coverage, a task-specific workflow for executable artifact generation, and runtime feedback analysis for debugging and refinement. NeTestLLM has been deployed in a production environment for several months, receiving positive feedback from domain experts. In experiments, NeTestLLM generated 4,632 test cases for OSPF, RIP, and BGP, covering 41 historical FRRouting bugs compared to 11 by current national standards. The process of generating executable artifacts also improves testing efficiency by a factor of 8.65x compared to manual methods. NeTestLLM provides the first practical LLM-powered solution for automated end-to-end testing of heterogeneous network protocols.

Figures (12)

• Figure 1: Network protocol testing workflow.
• Figure 2: A typical scenario of network protocol testing involving multiple stakeholders.
• Figure 3: Workflow of NeTestLLM. Modules marked with a robot icon are integrated with the LLM.
• Figure 4: Prompt template for module formation.
• Figure 5: Workflow of protocol-specific functional modeling.