Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predictive modeling and anomaly detection in large-scale web portals through the CAWAL framework

Ozkan Canay, Umit Kocabicak

TL;DR

This work introduces the CAWAL framework, a data-integration approach that fuses application logs with web analytics to produce enriched session and page-view datasets for predictive modeling and anomaly detection in Web Usage Mining. By eliminating preprocessing and leveraging multi-source data, CAWAL enables high-accuracy predictions (e.g., exiting behavior, last abandoned service, and service access) and effective server-load anomaly detection in a large-scale, multi-server web farm. The study demonstrates strong results across multiple models (Random Forest, Gradient Boosting, Isolation Forest) and provides a detailed data pipeline, from data collection to CSV generation for ML workflows. The framework offers practical benefits for portal optimization, performance monitoring, and security in enterprise-scale environments, while also outlining limitations and directions for future work across industries and regulatory contexts.

Abstract

This study presents an approach that uses session and page view data collected through the CAWAL framework, enriched through specialized processes, for advanced predictive modeling and anomaly detection in web usage mining (WUM) applications. Traditional WUM methods often rely on web server logs, which limit data diversity and quality. Integrating application logs with web analytics, the CAWAL framework creates comprehensive session and page view datasets, providing a more detailed view of user interactions and effectively addressing these limitations. This integration enhances data diversity and quality while eliminating the preprocessing stage required in conventional WUM, leading to greater process efficiency. The enriched datasets, created by cross-integrating session and page view data, were applied to advanced machine learning models, such as Gradient Boosting and Random Forest, which are known for their effectiveness in capturing complex patterns and modeling non-linear relationships. These models achieved over 92% accuracy in predicting user behavior and significantly improved anomaly detection capabilities. The results show that this approach offers detailed insights into user behavior and system performance metrics, making it a reliable solution for improving large-scale web portals' efficiency, reliability, and scalability.

Predictive modeling and anomaly detection in large-scale web portals through the CAWAL framework

TL;DR

This work introduces the CAWAL framework, a data-integration approach that fuses application logs with web analytics to produce enriched session and page-view datasets for predictive modeling and anomaly detection in Web Usage Mining. By eliminating preprocessing and leveraging multi-source data, CAWAL enables high-accuracy predictions (e.g., exiting behavior, last abandoned service, and service access) and effective server-load anomaly detection in a large-scale, multi-server web farm. The study demonstrates strong results across multiple models (Random Forest, Gradient Boosting, Isolation Forest) and provides a detailed data pipeline, from data collection to CSV generation for ML workflows. The framework offers practical benefits for portal optimization, performance monitoring, and security in enterprise-scale environments, while also outlining limitations and directions for future work across industries and regulatory contexts.

Abstract

This study presents an approach that uses session and page view data collected through the CAWAL framework, enriched through specialized processes, for advanced predictive modeling and anomaly detection in web usage mining (WUM) applications. Traditional WUM methods often rely on web server logs, which limit data diversity and quality. Integrating application logs with web analytics, the CAWAL framework creates comprehensive session and page view datasets, providing a more detailed view of user interactions and effectively addressing these limitations. This integration enhances data diversity and quality while eliminating the preprocessing stage required in conventional WUM, leading to greater process efficiency. The enriched datasets, created by cross-integrating session and page view data, were applied to advanced machine learning models, such as Gradient Boosting and Random Forest, which are known for their effectiveness in capturing complex patterns and modeling non-linear relationships. These models achieved over 92% accuracy in predicting user behavior and significantly improved anomaly detection capabilities. The results show that this approach offers detailed insights into user behavior and system performance metrics, making it a reliable solution for improving large-scale web portals' efficiency, reliability, and scalability.

Paper Structure

This paper contains 18 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Data preprocessing and session identification techniques
  4. Predictive modeling techniques
  5. Anomaly detection approaches
  6. Methodology
  7. Integration of the CAWAL framework
  8. Data flow of the model
  9. Data preparation and enrichment
  10. Generation of data files
  11. Feature engineering and model training
  12. Predictive analysis and findings
  13. Prediction of methods for leaving the system
  14. Prediction of the last abandoned service
  15. Prediction of access to a specific service
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Data flow and processing in the CAWAL model.
  • Figure 2: Transformation of web usage data into predictions.
  • Figure 3: Feature importance scores in the predictive model.
  • Figure 4: Visualization of anomalies based on page load times across seven servers in the web farm.