Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing

TL;DR

This work tackles the challenge of classifying massive, non-natural-language software logs from telecom network emulation into protocol-layer defects. It introduces a lightweight residual CNN capable of handling up to 200,000-character sequences and achieving a high F1 score (\(F1>0.9\)) while remaining edge-deployable (<3 MB). A seq2seq-derived character embedding initializes the CNN, and an overlapping-window strategy assesses long-context embeddings for multiple LLMs, which are shown to underperform relative to the tailored CNN in this domain. Through extensive experimentation, the CNN outperforms several LLM baselines (including LLaMA2-7B, Mixtral-8x7B, Flan-T5, BERT, and BigBird) in defect triage accuracy, while offering practical deployment benefits and generalizability to logs beyond telecommunications. The results highlight the value of domain-tailored architectures for industrial log analysis and inform production choices where context length and cost are critical factors.

Abstract

Software logs generated by sophisticated network emulators in the telecommunications industry, such as VIAVI TM500, are extremely complex, often comprising tens of thousands of text lines with minimal resemblance to natural language. Only specialised expert engineers can decipher such logs and troubleshoot defects in test runs. While AI offers a promising solution for automating defect triage, potentially leading to massive revenue savings for companies, state-of-the-art large language models (LLMs) suffer from significant drawbacks in this specialised domain. These include a constrained context window, limited applicability to text beyond natural language, and high inference costs. To address these limitations, we propose a compact convolutional neural network (CNN) architecture that offers a context window spanning up to 200,000 characters and achieves over 96% accuracy (F1>0.9) in classifying multifaceted software logs into various layers in the telecommunications protocol stack. Specifically, the proposed model is capable of identifying defects in test runs and triaging them to the relevant department, formerly a manual engineering process that required expert knowledge. We evaluate several LLMs; LLaMA2-7B, Mixtral 8x7B, Flan-T5, BERT and BigBird, and experimentally demonstrate their shortcomings in our specialized application. Despite being lightweight, our CNN significantly outperforms LLM-based approaches in telecommunications log classification while minimizing the cost of production. Our defect triaging AI model is deployable on edge devices without dedicated hardware and widely applicable across software logs in various industries.

Figures (9)

• Figure 1: Workflow demonstrating data preparation and architecture of CCOS model.
• Figure 2: Character length histograms of log files before (A) and after (B) cleaning.
• Figure 3: Proposed residual CNN architecture for software log classification.
• Figure 4: Distribution of classes in our full dataset (A) and test set (B).
• Figure 5: Training progress curves for the proposed residual CNN model. Note that the F1-score shown here is the micro average computed by obtaining the overall true positives, false positives, and false negatives across all classes.