Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification

TL;DR

The paper benchmarks small language models (SLMs) and small reasoning language models (SRLMs) on system log severity classification using real journalctl data, reframing severity labels as a diagnostic probe for runtime log understanding in digital twin pipelines. It compares zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting across a diverse model set, employing FAISS-based context retrieval with 768-d embeddings and a fixed prompt schema that enforces a single-digit output. Key findings show strong model stratification: RAG yields the largest performance gains for many models (e.g., up to $95.64\%$ with Qwen3-4B), but is not universally beneficial, as several SRLMs degrade or exhibit prohibitive latency under retrieval. The results highlight architectural and training-objective factors that govern retrieval integration, with implications for real-time, deployable log-analysis in DT systems and downstream tasks like root-cause analysis (RCA).

Abstract

System logs are crucial for monitoring and diagnosing modern computing infrastructure, but their scale and complexity require reliable and efficient automated interpretation. Since severity levels are predefined metadata in system log messages, having a model merely classify them offers limited standalone practical value, revealing little about its underlying ability to interpret system logs. We argue that severity classification is more informative when treated as a benchmark for probing runtime log comprehension rather than as an end task. Using real-world journalctl data from Linux production servers, we evaluate nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal strong stratification. Qwen3-4B achieves the highest accuracy at 95.64% with RAG, while Gemma3-1B improves from 20.25% under few-shot prompting to 85.28% with RAG. Notably, the tiny Qwen3-0.6B reaches 88.12% accuracy despite weak performance without retrieval. In contrast, several SRLMs, including Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B, degrade substantially when paired with RAG. Efficiency measurements further separate models: most Gemma and Llama variants complete inference in under 1.2 seconds per log, whereas Phi-4-Mini-Reasoning exceeds 228 seconds per log while achieving <10% accuracy. These findings suggest that (1) architectural design, (2) training objectives, and (3) the ability to integrate retrieved context under strict output constraints jointly determine performance. By emphasizing small, deployable models, this benchmark aligns with real-time requirements of digital twin (DT) systems and shows that severity classification serves as a lens for evaluating model competence and real-time deployability, with implications for root cause analysis (RCA) and broader DT integration.

Figures (5)

• Figure 1: Workflow for evaluating log-severity classification using SLMs and SRLMs. System logs are collected from real-world servers, processed through prompting strategies (zero-shot, few-shot, and RAG with FAISS), evaluated across various open-source models, and assessed by accuracy and inference speed.
• Figure 2: Retrieval-Augmented Generation (RAG) Pipeline for Syslog severity classification where incoming logs are embedded using a local LM Studio encoder (768-dimensional embeddings) and indexed with FAISS. At inference, each query log is embedded, and the top-$k$ nearest training vectors ($k$=5) are retrieved using L2 similarity and appended to a zero-shot prompt. The LLM then predicts the Syslog severity label conditioned on the retrieved examples.
• Figure 3: Performance of Qwen3-1.7B across retrieval depths. Reducing $k$ from 5 to 1 decreases latency but does not improve accuracy, indicating limited retrieval integration at this model scale.
• Figure 4: Performance of evaluated models under zero-shot, few-shot, and RAG prompting. Retrieval yields the strongest overall gains, with notable variation across architectures.
• Figure 5: Average per-sample inference time. Several models approach real-time latency, though performance varies significantly by scale and prompting mode.