AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model

TL;DR

AdaptiveLog addresses the cost-inefficiency of large language models in automated log analysis by introducing a collaborative framework with a small language model. It uses Bayesian uncertainty estimation to route uncertain cases to an LLM and leverages Error-Case Reasoning Enhancement to improve LLM reasoning with retrieved error-prone cases. Across six software- and network-domain tasks, AdaptiveLog achieves state-of-the-art accuracy while reducing LLM usage by up to 73% and showing strong performance in low-resource and transfer-learning scenarios. The approach demonstrates practical impact by offering a scalable, cost-efficient solution for production log analysis that can adapt to different SLMs and vendors.

Abstract

Automated log analysis is crucial to ensure high availability and reliability of complex systems. The advent of LLMs in NLP has ushered in a new era of language model-driven automated log analysis, garnering significant interest. Within this field, two primary paradigms based on language models for log analysis have become prominent. Small Language Models (SLMs) follow the pre-train and fine-tune paradigm, focusing on the specific log analysis task through fine-tuning on supervised datasets. On the other hand, LLMs following the in-context learning paradigm, analyze logs by providing a few examples in prompt contexts without updating parameters. Despite their respective strengths, we notice that SLMs are more cost-effective but less powerful, whereas LLMs with large parameters are highly powerful but expensive and inefficient. To trade-off between the performance and inference costs of both models in automated log analysis, this paper introduces an adaptive log analysis framework known as AdaptiveLog, which effectively reduces the costs associated with LLM while ensuring superior results. This framework collaborates an LLM and a small language model, strategically allocating the LLM to tackle complex logs while delegating simpler logs to the SLM. Specifically, to efficiently query the LLM, we propose an adaptive selection strategy based on the uncertainty estimation of the SLM, where the LLM is invoked only when the SLM is uncertain. In addition, to enhance the reasoning ability of the LLM in log analysis tasks, we propose a novel prompt strategy by retrieving similar error-prone cases as the reference, enabling the model to leverage past error experiences and learn solutions from these cases. Extensive experiments demonstrate that AdaptiveLog achieves state-of-the-art results across different tasks, elevating the overall accuracy of log analysis while maintaining cost efficiency.

Figures (16)

• Figure 1: Different log analysis frameworks for anomaly detection with language models, where SLMs are efficient but low performance, and LLMs are inefficient but high performance. Our proposed framework combines the advantages of both adequately.
• Figure 2: The framework of AdaptiveLog, which comprises three key steps and a preparation stage. It first analyzes logs with the SLM and adaptively chooses to invoke the LLM based on the uncertainty of the SLM. The preparation stage constructs an error-prone case database to enhance the reasoning of the LLM.
• Figure 3: The prompt template example of ECR on the anomaly detection task, where the error-prone cases including reasoning process and potential pitfalls are provided in the prompt to improve the reasoning of LLMs.
• Figure 4: Results in the low-resource scenario.
• Figure 5: Results in the transfer learning scenario. Left side of $\rightarrow$ indicates the source dataset for training and right side indicates the target dataset for testing.