CellularSpecSec-Bench: A Staged Benchmark for Evidence-Grounded Interpretation and Security Reasoning over 3GPP Specifications

TL;DR

This work addresses the challenge of trustworthy interpretation and security reasoning over dense, cross-referenced 3GPP specifications by introducing CellSpecSec-ARI, a three-stage Adapt-Retrieve-Integrate framework comprising SpecFusion, SpecRAG, and SpecReasoning. It also delivers CellularSpecSec-Bench, a staged, expert-verified benchmark built from Release-17 specifications and supplementary datasets to evaluate clause-local understanding, evidence-grounded answering, and cross-clause security reasoning. Empirical results show substantial gains from modular adaptations and retrieval-grounded reasoning, with robust performance on external tasks and detailed error analyses guiding future improvements in evidence reliability and reasoning completeness. The framework and benchmark provide a reproducible foundation for measuring progress in standards-driven interpretation and security analysis of cellular network protocols.

Abstract

Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. While large language models (LLMs) have demonstrated the capability for reading technical documents, cellular specifications impose unique challenges: faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures. To address these challenges, we propose CellSpecSec-ARI, a unified Adapt-Retrieve-Integrate framework for systematic understanding and standard-driven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed high-quality datasets with expert-verified and corrected subsets from prior open-source resources. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in the cellular network security domain.

Figures (16)

• Figure 1: Overview of CellSpecSec-ARI and CellularSpecSec-Bench
• Figure 2: Clause-level text extraction
• Figure 3: Figure and table identification extraction
• Figure 4: Figure content extraction
• Figure 5: Table Content Extraction