LADFA: A Framework of Using Large Language Models and Retrieval-Augmented Generation for Personal Data Flow Analysis in Privacy Policies
Haiyue Yuan, Nikolay Matyunin, Ali Raza, Shujun Li
TL;DR
This work tackles the problem of interpreting privacy policies by automating the extraction of comprehensive personal data flows. It introduces LADFA, an end-to-end framework that combines LLMs with retrieval-augmented generation and a domain-specific knowledge base to identify data categories, data consumer types, processing purposes, and processing methods, and to construct data-flow graphs. A three-part pipeline (pre-processor, LLM-based processor with LLM-RAG agents, and post-processor) enables segmentation, structured extraction, and graph-based analysis, demonstrated in a case study of ten automotive OEM apps. Results show strong expert agreement on data types and data flows, with actionable insights on cross-border transfers, third-party dependencies, and policy transparency, while acknowledging limitations in ground-truth data, KB vagueness, and segmentation strategies. The work argues for broader applicability beyond privacy policies and highlights future opportunities to extend datasets, refine knowledge bases, and integrate with other graph tools for richer governance analyses.
Abstract
Privacy policies help inform people about organisations' personal data processing practices, covering different aspects such as data collection, data storage, and sharing of personal data with third parties. Privacy policies are often difficult for people to fully comprehend due to the lengthy and complex legal language used and inconsistent practices across different sectors and organisations. To help conduct automated and large-scale analyses of privacy policies, many researchers have studied applications of machine learning and natural language processing techniques, including large language models (LLMs). While a limited number of prior studies utilised LLMs for extracting personal data flows from privacy policies, our approach builds on this line of work by combining LLMs with retrieval-augmented generation (RAG) and a customised knowledge base derived from existing studies. This paper presents the development of LADFA, an end-to-end computational framework, which can process unstructured text in a given privacy policy, extract personal data flows and construct a personal data flow graph, and conduct analysis of the data flow graph to facilitate insight discovery. The framework consists of a pre-processor, an LLM-based processor, and a data flow post-processor. We demonstrated and validated the effectiveness and accuracy of the proposed approach by conducting a case study that involved examining ten selected privacy policies from the automotive industry. Moreover, it is worth noting that LADFA is designed to be flexible and customisable, making it suitable for a range of text-based analysis tasks beyond privacy policy analysis.