Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Generation of Accurate Privacy Captions From Android Source Code Using Large Language Models

Vijayanta Jain, Sepideh Ghanavati, Sai Teja Peddinti, Collin McMillan

TL;DR

This work tackles the problem of generating accurate privacy captions directly from Android source code to support privacy notices and regulatory compliance. It introduces PCapGen, a three-component pipeline (Identifier, Extractor, Generator) that builds large-scale code context via taint-path extraction, decompiles APKs to obtain source code, and uses large language models with in-context learning to produce concise, complete captions. The authors propose a novel heuristics-based taint analysis to uncover unknown sinks, create a 442-sample dataset of developer-annotated captions, and evaluate across three model configurations (GPT-4, Claude Opus, DeepSeek) using LLM-judge models and privacy experts. Results show PCapGen, especially PCapGen_Claude, generates captions that are semantically aligned with baselines but are rated higher on accuracy, conciseness, and completeness, with strong preferences from judges and sizeable expert support, demonstrating practical potential for automated privacy notices.

Abstract

Privacy captions are short sentences that succinctly describe what personal information is used, how it is used, and why, within an app. These captions can be utilized in various notice formats, such as privacy policies, app rationales, and app store descriptions. However, inaccurate captions may mislead users and expose developers to regulatory fines. Existing approaches to generating privacy notices or just privacy captions include using questionnaires, templates, static analysis, or machine learning. However, these approaches either rely heavily on developers' inputs and thus strain their efforts, use limited source code context, leading to the incomplete capture of app privacy behaviors, or depend on potentially inaccurate privacy policies as a source for creating notices. In this work, we address these limitations by developing Privacy Caption Generator (PCapGen), an approach that - i) automatically identifies and extracts large and precise source code context that implements privacy behaviors in an app, ii) uses a Large Language Model (LLM) to describe coarse- and fine-grained privacy behaviors, and iii) generates accurate, concise, and complete privacy captions to describe the privacy behaviors of the app. Our evaluation shows PCapGen generates concise, complete, and accurate privacy captions as compared to the baseline approach. Furthermore, privacy experts choose PCapGen captions at least 71\% of the time, whereas LLMs-as-judge prefer PCapGen captions at least 76\% of the time, indicating strong performance of our approach.

Automated Generation of Accurate Privacy Captions From Android Source Code Using Large Language Models

TL;DR

This work tackles the problem of generating accurate privacy captions directly from Android source code to support privacy notices and regulatory compliance. It introduces PCapGen, a three-component pipeline (Identifier, Extractor, Generator) that builds large-scale code context via taint-path extraction, decompiles APKs to obtain source code, and uses large language models with in-context learning to produce concise, complete captions. The authors propose a novel heuristics-based taint analysis to uncover unknown sinks, create a 442-sample dataset of developer-annotated captions, and evaluate across three model configurations (GPT-4, Claude Opus, DeepSeek) using LLM-judge models and privacy experts. Results show PCapGen, especially PCapGen_Claude, generates captions that are semantically aligned with baselines but are rated higher on accuracy, conciseness, and completeness, with strong preferences from judges and sizeable expert support, demonstrating practical potential for automated privacy notices.

Abstract

Privacy captions are short sentences that succinctly describe what personal information is used, how it is used, and why, within an app. These captions can be utilized in various notice formats, such as privacy policies, app rationales, and app store descriptions. However, inaccurate captions may mislead users and expose developers to regulatory fines. Existing approaches to generating privacy notices or just privacy captions include using questionnaires, templates, static analysis, or machine learning. However, these approaches either rely heavily on developers' inputs and thus strain their efforts, use limited source code context, leading to the incomplete capture of app privacy behaviors, or depend on potentially inaccurate privacy policies as a source for creating notices. In this work, we address these limitations by developing Privacy Caption Generator (PCapGen), an approach that - i) automatically identifies and extracts large and precise source code context that implements privacy behaviors in an app, ii) uses a Large Language Model (LLM) to describe coarse- and fine-grained privacy behaviors, and iii) generates accurate, concise, and complete privacy captions to describe the privacy behaviors of the app. Our evaluation shows PCapGen generates concise, complete, and accurate privacy captions as compared to the baseline approach. Furthermore, privacy experts choose PCapGen captions at least 71\% of the time, whereas LLMs-as-judge prefer PCapGen captions at least 76\% of the time, indicating strong performance of our approach.
Paper Structure (32 sections, 4 figures, 8 tables)

This paper contains 32 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed PCapGen Approach
  4. Identifier: Identifying Taint Paths
  5. Extractor: Extracting Source Code
  6. Generator: Generating Privacy Captions
  7. Baseline Captions, Survey, and Dataset
  8. Extracting Android Source Code Samples
  9. Baseline Approach to Create Privacy Captions
  10. Survey Design
  11. Pilot Study
  12. Participants and Baseline Caption Dataset
  13. Evaluation Experiments
  14. RQ1: Comparing PCapGen Configurations
  15. RQ2: Comparing Baseline and PCapGen
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Overview of our PCapGen describing the key components and steps to generate privacy captions.
  • Figure 2: An example of a taint path describing the flow of location information from the source method getLoc being saved to a log file in the sink method saveToLog.
  • Figure 3: User Interface of Final Annotation Tool.
  • Figure 4: Human evaluation ratings for PCapGen_Claude and Baseline across three dimensions.