Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models in detecting Secrets in Android Apps

Marco Alecci, Jordan Samhi, Tegawendé F. Bissyandé, Jacques Klein

TL;DR

SecretLoc introduces an LLM-based approach to detect hardcoded secrets in Android apps without relying on predefined patterns. By analyzing XML strings and code strings through a four-phase pipeline (A1/A2, B1/B2) and leveraging contextual reasoning, it outperforms regex, static, and ML baselines on a 5135-app benchmark and uncovers thousands of additional secrets. Real-world evaluation on 5000 Google Play apps shows a substantial prevalence of secrets (42.5%), with remediation by some developers after disclosure, underscoring both the defensive potential and dual-use risks of LLM-based secret detection. The work highlights the need for proactive secret management in the mobile ecosystem and calls for responsible use to prevent misuse of such powerful analysis tools.

Abstract

Mobile apps often embed authentication secrets, such as API keys, tokens, and client IDs, to integrate with cloud services. However, developers often hardcode these credentials into Android apps, exposing them to extraction through reverse engineering. Once compromised, adversaries can exploit secrets to access sensitive data, manipulate resources, or abuse APIs, resulting in significant security and financial risks. Existing detection approaches, such as regex-based analysis, static analysis, and machine learning, are effective for identifying known patterns but are fundamentally limited: they require prior knowledge of credential structures, API signatures, or training data. In this paper, we propose SecretLoc, an LLM-based approach for detecting hardcoded secrets in Android apps. SecretLoc goes beyond pattern matching; it leverages contextual and structural cues to identify secrets without relying on predefined patterns or labeled training sets. Using a benchmark dataset from the literature, we demonstrate that SecretLoc detects secrets missed by regex-, static-, and ML-based methods, including previously unseen types of secrets. In total, we discovered 4828 secrets that were undetected by existing approaches, discovering more than 10 "new" types of secrets, such as OpenAI API keys, GitHub Access Tokens, RSA private keys, and JWT tokens, and more. We further extend our analysis to newly crawled apps from Google Play, where we uncovered and responsibly disclosed additional hardcoded secrets. Across a set of 5000 apps, we detected secrets in 2124 apps (42.5%), several of which were confirmed and remediated by developers after we contacted them. Our results reveal a dual-use risk: if analysts can uncover these secrets with LLMs, so can attackers. This underscores the urgent need for proactive secret management and stronger mitigation practices across the mobile ecosystem.

Evaluating Large Language Models in detecting Secrets in Android Apps

TL;DR

SecretLoc introduces an LLM-based approach to detect hardcoded secrets in Android apps without relying on predefined patterns. By analyzing XML strings and code strings through a four-phase pipeline (A1/A2, B1/B2) and leveraging contextual reasoning, it outperforms regex, static, and ML baselines on a 5135-app benchmark and uncovers thousands of additional secrets. Real-world evaluation on 5000 Google Play apps shows a substantial prevalence of secrets (42.5%), with remediation by some developers after disclosure, underscoring both the defensive potential and dual-use risks of LLM-based secret detection. The work highlights the need for proactive secret management in the mobile ecosystem and calls for responsible use to prevent misuse of such powerful analysis tools.

Abstract

Mobile apps often embed authentication secrets, such as API keys, tokens, and client IDs, to integrate with cloud services. However, developers often hardcode these credentials into Android apps, exposing them to extraction through reverse engineering. Once compromised, adversaries can exploit secrets to access sensitive data, manipulate resources, or abuse APIs, resulting in significant security and financial risks. Existing detection approaches, such as regex-based analysis, static analysis, and machine learning, are effective for identifying known patterns but are fundamentally limited: they require prior knowledge of credential structures, API signatures, or training data. In this paper, we propose SecretLoc, an LLM-based approach for detecting hardcoded secrets in Android apps. SecretLoc goes beyond pattern matching; it leverages contextual and structural cues to identify secrets without relying on predefined patterns or labeled training sets. Using a benchmark dataset from the literature, we demonstrate that SecretLoc detects secrets missed by regex-, static-, and ML-based methods, including previously unseen types of secrets. In total, we discovered 4828 secrets that were undetected by existing approaches, discovering more than 10 "new" types of secrets, such as OpenAI API keys, GitHub Access Tokens, RSA private keys, and JWT tokens, and more. We further extend our analysis to newly crawled apps from Google Play, where we uncovered and responsibly disclosed additional hardcoded secrets. Across a set of 5000 apps, we detected secrets in 2124 apps (42.5%), several of which were confirmed and remediated by developers after we contacted them. Our results reveal a dual-use risk: if analysts can uncover these secrets with LLMs, so can attackers. This underscores the urgent need for proactive secret management and stronger mitigation practices across the mobile ecosystem.
Paper Structure (18 sections, 5 figures, 8 tables)

This paper contains 18 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Approach
  4. Considerations about Phase [B1]
  5. Implementation Details
  6. Evaluation
  7. RQ1: Comparison with Existing Tools
  8. Analysis of Previously “Clean” Apps
  9. RQ2: Accuracy of Assigned Labels
  10. RQ3: Impact of the Underlying LLM Choice
  11. RQ4: Impact of Adding Code Context in the First Phase
  12. RQ5: Performance on Real-World Apps
  13. Disclosure to Developers
  14. Discussion
  15. Limitations
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Approach Overview.
  • Figure 2: Comparison between existing tools and SecretLoc.
  • Figure 3: Mapping between benchmark categories (derived from regex/API matches in Li et al. li2024automaticallydetectingcheckedinsecrets) and the labels assigned by SecretLoc.
  • Figure 4: Comparison of the performance of different LLMs integrated in SecretLoc.
  • Figure 5: Overlap of secrets detected in code by the original and contextual variants of SecretLoc.