Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Complexity: Intelligent Pattern Exploration with CHPDA (Context Aware Hybrid Pattern Detection Algorithm)

Lokesh Koli, Shubham Kalra, Karanpreet Singh

TL;DR

The paper tackles the challenge of detecting PII/PHI across heterogeneous data sources under regulatory constraints. It proposes CHPDA, a context-aware hybrid pattern detection framework that fuses regex-based matching with AI-powered NER and exact-match using Aho-Corasick, augmented by proximity scoring and threshold filtering for robust, real-time detection. Empirical results show Google RE2 offers a favorable speed/memory/accuracy balance among regex engines, while Aho-Corasick provides superior exact-match performance; integrating AI with regex yields the best F1 score of 91.6%, with strong device-scale performance. Limitations include multilingual support gaps and the need for ongoing pattern updates, with future work targeting broader language coverage and DSPM/DLP integration.

Abstract

Detecting sensitive data such as Personally Identifiable Information (PII) and Protected Health Information (PHI) is critical for data security platforms. This study evaluates regex-based pattern matching algorithms and exact-match search techniques to optimize detection speed, accuracy, and scalability. Our benchmarking results indicate that Google RE2 provides the best balance of speed (10-15 ms/MB), memory efficiency (8-16 MB), and accuracy (99.5%) among regex engines, outperforming PCRE while maintaining broader hardware compatibility than Hyperscan. For exact matching, Aho-Corasick demonstrated superior performance (8 ms/MB) and scalability for large datasets. Performance analysis revealed that regex processing time scales linearly with dataset size and pattern complexity. A hybrid AI + Regex approach achieved the highest F1 score (91. 6%) by improving recall and minimizing false positives. Device benchmarking confirmed that our solution maintains efficient CPU and memory usage on both high-performance and mid-range systems. Despite its effectiveness, challenges remain, such as limited multilingual support and the need for regular pattern updates. Future work should focus on expanding language coverage, integrating data security and privacy management (DSPM) with data loss prevention (DLP) tools, and enhancing regulatory compliance for broader global adoption.

Decoding Complexity: Intelligent Pattern Exploration with CHPDA (Context Aware Hybrid Pattern Detection Algorithm)

TL;DR

The paper tackles the challenge of detecting PII/PHI across heterogeneous data sources under regulatory constraints. It proposes CHPDA, a context-aware hybrid pattern detection framework that fuses regex-based matching with AI-powered NER and exact-match using Aho-Corasick, augmented by proximity scoring and threshold filtering for robust, real-time detection. Empirical results show Google RE2 offers a favorable speed/memory/accuracy balance among regex engines, while Aho-Corasick provides superior exact-match performance; integrating AI with regex yields the best F1 score of 91.6%, with strong device-scale performance. Limitations include multilingual support gaps and the need for ongoing pattern updates, with future work targeting broader language coverage and DSPM/DLP integration.

Abstract

Detecting sensitive data such as Personally Identifiable Information (PII) and Protected Health Information (PHI) is critical for data security platforms. This study evaluates regex-based pattern matching algorithms and exact-match search techniques to optimize detection speed, accuracy, and scalability. Our benchmarking results indicate that Google RE2 provides the best balance of speed (10-15 ms/MB), memory efficiency (8-16 MB), and accuracy (99.5%) among regex engines, outperforming PCRE while maintaining broader hardware compatibility than Hyperscan. For exact matching, Aho-Corasick demonstrated superior performance (8 ms/MB) and scalability for large datasets. Performance analysis revealed that regex processing time scales linearly with dataset size and pattern complexity. A hybrid AI + Regex approach achieved the highest F1 score (91. 6%) by improving recall and minimizing false positives. Device benchmarking confirmed that our solution maintains efficient CPU and memory usage on both high-performance and mid-range systems. Despite its effectiveness, challenges remain, such as limited multilingual support and the need for regular pattern updates. Future work should focus on expanding language coverage, integrating data security and privacy management (DSPM) with data loss prevention (DLP) tools, and enhancing regulatory compliance for broader global adoption.

Paper Structure

This paper contains 27 sections, 11 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Regex-Based Detection
  4. AI-Driven Named Entity Recognition
  5. Hybrid AI + Regex Approaches
  6. Comparative Benchmarking and Efficiency
  7. Contributions of This Work
  8. Materials and methods
  9. Proposed system
  10. Regex
  11. Rescan with AI Model
  12. Exact Match: Optimized with Aho-Corasick algorithm
  13. Patterns:
  14. Text:
  15. Testing authentication functions
  16. ...and 12 more sections

Figures (4)

  • Figure 1: proposed System
  • Figure 3: Trie Data Structure Representation
  • Figure 4: Folder Classification Time vs. Data Size for Different Regex Sets
  • Figure 5: Total Matches and Missed Matches vs. File Size