LogDB: Multivariate Log-based Failure Diagnosis for Distributed Databases (Extended from MultiLog)

TL;DR

LogDB targets failure diagnosis in distributed databases by leveraging multivariate log information across cluster nodes. It embeds per-node logs using sequential, quantitative, and semantic features, then uses a self-attention-enhanced LSTM to produce node-level anomaly matrices, which are fused with a Variational Autoencoder and classified by a CNN to identify cluster-wide fault types. The approach addresses multi-node complexity, incomplete per-node logs, and diverse anomaly types, demonstrating robust performance on Apache IoTDB across multiple workloads and fault scenarios. This work offers practical diagnostic insight for large-scale databases and points toward multimodal, cross-node monitoring as a path to more effective AIOps for distributed systems.

Abstract

Distributed databases, as the core infrastructure software for internet applications, play a critical role in modern cloud services. However, existing distributed databases frequently experience system failures and performance degradation, often leading to significant economic losses. Log data, naturally generated within systems, can effectively reflect internal system states. In practice, operators often manually inspect logs to monitor system behavior and diagnose anomalies, a process that is labor-intensive and costly. Although various log-based failure diagnosis methods have been proposed, they are generally not tailored for database systems and fail to fully exploit the internal characteristics and distributed nature of these systems. To address this gap, we propose LogDB, a log-based failure diagnosis method specifically designed for distributed databases. LogDB extracts and compresses log features at each database node and then aggregates these features at the master node to diagnose cluster-wide anomalies. Experiments conducted on the open-source distributed database system Apache IoTDB demonstrate that LogDB achieves robust failure diagnosis performance across different workloads and a variety of anomaly types.

Figures (5)

• Figure 1: Common Workflow of Log-based failure diagnosis
• Figure 2: Architecture of LogDB
• Figure 3: Anomaly Injection Procedure
• Figure 4: Performance of LogDB under different autoencoder parameters.
• Figure 5: Performance of LogDB under different input settings.