Trust, But Verify: An Empirical Evaluation of AI-Generated Code for SDN Controllers
Felipe Avencourt Soares, Muriel F. Franco, Eder J. Scheid, Lisandro Z. Granville
TL;DR
This study evaluates AI-generated SDN controller code produced by four tools (ChatGPT, Copilot, DeepSeek, BlackBox.ai) across three tasks of increasing complexity, using zero-shot and few-shot prompts and testing in Mininet. All tools could produce functional POX controllers, with ChatGPT and DeepSeek delivering higher code quality and modularity than Copilot and BlackBox.ai, especially under few-shot prompting. The results highlight the value of few-shot prompts for robustness and readability while underscoring the need for human oversight and design scoping in rapidly evolving AI tooling. Practically, the work demonstrates both viability and limitations of AI-generated network programming, and points to future work on broader platforms (e.g., P4) and additional models to better understand AI-assisted SDN development.
Abstract
Generative Artificial Intelligence (AI) tools have been used to generate human-like content across multiple domains (e.g., sound, image, text, and programming). However, their reliability in terms of correctness and functionality in novel contexts such as programmable networks remains unclear. Hence, this paper presents an empirical evaluation of the source code of a POX controller generated by different AI tools, namely ChatGPT, Copilot, DeepSeek, and BlackBox.ai. To evaluate such a code, three networking tasks of increasing complexity were defined and for each task, zero-shot and few-shot prompting techniques were input to the tools. Next, the output code was tested in emulated network topologies with Mininet and analyzed according to functionality, correctness, and the need for manual fixes. Results show that all evaluated models can produce functional controllers. However, ChatGPT and DeepSeek exhibited higher consistency and code quality, while Copilot and BlackBox.ai required more adjustments.