ContraLog: Log File Anomaly Detection with Contrastive Learning and Masked Language Modeling

TL;DR

ContraLog tackles log anomaly detection without traditional parsing by learning to predict continuous log-message embeddings. A two-tower architecture (MessageEncoder and SequenceEncoder) is trained with masked-embedding prediction and a symmetric InfoNCE loss to align context-based predictions with observed embeddings. At inference, both contextual and point anomaly signals are computed and fused into a robust sequence-level score, enabling effective detection across diverse datasets (HDFS, BGL, Thunderbird) with minimal preprocessing and efficient caching. The results demonstrate that embedding-based predictions can outperform discrete template prediction in many settings and highlight the practical impact for scalable, parser-free log analysis in dynamic systems.

Abstract

Log files record computational events that reflect system state and behavior, making them a primary source of operational insights in modern computer systems. Automated anomaly detection on logs is therefore critical, yet most established methods rely on log parsers that collapse messages into discrete templates, discarding variable values and semantic content. We propose ContraLog, a parser-free and self-supervised method that reframes log anomaly detection as predicting continuous message embeddings rather than discrete template IDs. ContraLog combines a message encoder that produces rich embeddings for individual log messages with a sequence encoder to model temporal dependencies within sequences. The model is trained with a combination of masked language modeling and contrastive learning to predict masked message embeddings based on the surrounding context. Experiments on the HDFS, BGL, and Thunderbird benchmark datasets empirically demonstrate effectiveness on complex datasets with diverse log messages. Additionally, we find that message embeddings generated by ContraLog carry meaningful information and are predictive of anomalies even without sequence context. These results highlight embedding-level prediction as an approach for log anomaly detection, with potential applicability to other event sequences.

Figures (13)

• Figure 1: Overview of ContraLog. (1) Log messages of a sequence are individually embedded. Then, random embeddings are masked, and the SequenceEncoder predicts the original masked representations based on the context they occur in. (2) Both encoders are trained end-to-end using contrastive learning. We aim to maximize the similarity between the observed and the predicted embedding. (3) Anomalies are evaluated by two criteria. The point anomaly score estimates how similar a new embedding is to those seen during training. The contextual anomaly score measures how close the predicted embedding is to the observed one. (4) At inference, each message in a sequence is masked one after another, resulting in two anomaly scores for each message. For anomaly detection, we aggregate the scores into sequence-level features, which are standardized with robust z-scores and combined via the $L_2$ norm into a single anomaly score.
• Figure 2: Visualization of the embedding spaces for messages from the HDFS, BGL, and Thunderbird datasets using UMAP dimensionality reduction UMAP. Orange embeddings originate from abnormal sequences and blue embeddings from normal sequences.
• Figure 3: Comparison of log message compression rates between a custom Byte Pair Encoding (BPE) tokenizer and the pre-fitted BERT-Base Uncased BERT tokenizer across different datasets.
• Figure 4: F1 scores achieved on the test set when using different percentiles of the anomaly score distribution on the training set as a threshold.
• Figure 5: Receiver Operating Characteristic (ROC) curves for different datasets.