Adaptive and Efficient Log Parsing as a Cloud Service
Zeyan Li, Jie Song, Tieying Zhang, Tao Yang, Xiongjun Ou, Yingjie Ye, Pengfei Duan, Muchen Lin, Jianjun Chen
TL;DR
This work tackles the challenge of scalable, adaptable log parsing for large-scale cloud environments. It introduces ByteBrain-LogParser, a two-phase framework combining offline hierarchical clustering with online matching to support real-time precision adjustments at query time. Key innovations include positional similarity distance, saturation-based clustering, deduplication, and hash encoding, which together deliver near-SOTA accuracy at unprecedented throughput (e.g., 229k logs/sec) and a small model footprint. Industrial deployment in Volcano Engine’s Torch Log Service demonstrates practical effectiveness, showing high throughput, low end-to-end latency, and dynamic template management in production. The approach offers a cost-effective, cloud-native solution for automated log analysis with robust performance across diverse log sources and scales.
Abstract
Logs are a critical data source for cloud systems, enabling advanced features like monitoring, alerting, and root cause analysis. However, the massive scale and diverse formats of unstructured logs pose challenges for adaptable, efficient, and accurate parsing methods. This paper introduces ByteBrain-LogParser, an innovative log parsing framework designed specifically for cloud environments. ByteBrain-LogParser employs a hierarchical clustering algorithm to allow real-time precision adjustments, coupled with optimizations such as positional similarity distance, deduplication, and hash encoding to enhance performance. Experiments on large-scale datasets show that it processes 229,000 logs per second on average, achieving an 840% speedup over the fastest baseline while maintaining accuracy comparable to state-of-the-art methods. Real-world evaluations further validate its efficiency and adaptability, demonstrating its potential as a robust cloud-based log parsing solution.