Can Large Language Models Detect Real-World Android Software Compliance Violations?

TL;DR

CompliBench introduces a realism-first benchmark that binds Android code evidence to native privacy articles across LGPD, PDPA, and PIPEDA to evaluate LLMs on compliance detection. It formalizes two auditor-aligned tasks—cross-granularity retrieval and fragment-level judgment—and couples them with a stability-centric evaluation framework (SGS, RCS, CRGS, OCS) to reveal robustness across granularity and jurisdiction. Empirical results across six models show strong retrieval or judgment in isolation but notable brittleness in end-to-end performance, underscoring the need for balanced retriever–judge systems and cross-law generalization. The work provides actionable tooling and open data/prompts to advance regulation-aware software engineering and opens avenues for incorporating richer static/dynamic priors and broader statutory coverage in future iterations.

Abstract

The rapid development of Large Language Models (LLMs) has transformed software engineering, showing promise in tasks like code generation, bug detection, and compliance checking. However, current models struggle to detect compliance violations in Android applications across diverse legal frameworks. We propose \emph{CompliBench}, a novel evaluation framework for assessing LLMs' ability to detect compliance violations under regulations like LGPD, PDPA, and PIPEDA. The framework defines two tasks: Task 1 evaluates \emph{retrieval and localization} at file, module, and line granularities, and Task 2 assesses \emph{multi-label judgment} for code snippets. These tasks mirror the audit process, where auditors locate problematic code and determine implicated provisions. Traditional metrics fail to capture important aspects like cross-granularity stability and jurisdictional consistency. Thus, we introduce stability-aware composites (SGS, RCS, CRGS, and OCS) for a more comprehensive assessment. Experiments with six models, including GPT-4O and Claude-3.5, show \emph{CompliBench} improves compliance detection, with Claude-3.5-sonnet-20241022 achieving the highest OCS score (0.3295), and Gemini-2.5-pro the lowest (0.0538). This work demonstrates \emph{CompliBench}'s potential for improving LLM performance in compliance tasks and provides a foundation for future tools aligned with data protection standards. Our project is available at https://github.com/Haoyi-Zhang/CompliBench.

Figures (6)

• Figure 1: CompliBench overview. Android repositories and statutory articles are transformed into two evaluation tracks: Task 1 (retrieval at file/module/line granularity) and Task 2 (multi-label classification at the snippet level). Standard metrics feed into stability-aware composites (SGS, RCS, CRGS, OCS), yielding cross-granularity and cross-jurisdiction insights. The figure adopts a staged layout (Repos$\rightarrow$Annotation & Normalization$\rightarrow$Schema & Tasks$\rightarrow$Metrics$\rightarrow$Insights) with branching evaluation tracks and consolidated ranking diagnostics.
• Figure 2: Task Design Overview. The figure illustrates the workflow for Task 1 (retrieval and localization) and Task 2 (multi-label judgment).
• Figure 3: Task 1 (file level). Radar plots per model. Axes: Acc@1, Acc@5, R–Precision, MRR, MAP, nDCG@5. Larger, more regular polygons indicate stronger and more balanced retrieval at the file level.
• Figure 4: Task 1 (module level). Compared with file level, many models contract on MRR/Acc@1 while preserving Acc@5, revealing difficulties in precise ordering at component granularity.
• Figure 5: Task 1 (line level). Polygons often dilate relative to coarser levels, especially on ranking–sensitive axes (MRR, MAP, nDCG@5), reflecting the benefit of precise anchors.