Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PrivLEX: Detecting legal concepts in images through Vision-Language Models

Darya Baranouskaya, Andrea Cavallaro

TL;DR

PrivLEX tackles the problem of identifying potential personal data leaks in images by grounding predictions in legally defined personal data concepts from the DPV-PD taxonomy. It employs a label-free Concept Bottleneck Model with zero-shot concept detection from Vision-Language Models, computing concept scores $c_{ij}$ via $c_{ij} = \frac{I_i \cdot T_j}{\|I_i\| \|T_j\|}$, normalizing to $[0,1]$, and using a sparse Logistic Regression classifier to predict privacy while providing interpretable concept-level explanations. The method achieves state-of-the-art performance among interpretable privacy classifiers on PrivacyAlert and VISPR, while enabling analysis of concept-level contributions and dataset biases. The work demonstrates the practicality of legally grounded, interpretable privacy reasoning in images and outlines avenues for improvement, such as leveraging the DPV-PD hierarchy and incorporating OCR for document understanding.

Abstract

We present PrivLEX, a novel image privacy classifier that grounds its decisions in legally defined personal data concepts. PrivLEX is the first interpretable privacy classifier aligned with legal concepts that leverages the recognition capabilities of Vision-Language Models (VLMs). PrivLEX relies on zero-shot VLM concept detection to provide interpretable classification through a label-free Concept Bottleneck Model, without requiring explicit concept labels during training. We demonstrate PrivLEX's ability to identify personal data concepts that are present in images. We further analyse the sensitivity of such concepts as perceived by human annotators of image privacy datasets.

PrivLEX: Detecting legal concepts in images through Vision-Language Models

TL;DR

PrivLEX tackles the problem of identifying potential personal data leaks in images by grounding predictions in legally defined personal data concepts from the DPV-PD taxonomy. It employs a label-free Concept Bottleneck Model with zero-shot concept detection from Vision-Language Models, computing concept scores via , normalizing to , and using a sparse Logistic Regression classifier to predict privacy while providing interpretable concept-level explanations. The method achieves state-of-the-art performance among interpretable privacy classifiers on PrivacyAlert and VISPR, while enabling analysis of concept-level contributions and dataset biases. The work demonstrates the practicality of legally grounded, interpretable privacy reasoning in images and outlines avenues for improvement, such as leveraging the DPV-PD hierarchy and incorporating OCR for document understanding.

Abstract

We present PrivLEX, a novel image privacy classifier that grounds its decisions in legally defined personal data concepts. PrivLEX is the first interpretable privacy classifier aligned with legal concepts that leverages the recognition capabilities of Vision-Language Models (VLMs). PrivLEX relies on zero-shot VLM concept detection to provide interpretable classification through a label-free Concept Bottleneck Model, without requiring explicit concept labels during training. We demonstrate PrivLEX's ability to identify personal data concepts that are present in images. We further analyse the sensitivity of such concepts as perceived by human annotators of image privacy datasets.
Paper Structure (9 sections, 2 equations, 4 figures, 3 tables)

This paper contains 9 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Detecting Legal Concepts in Images
  4. Experiments
  5. Experimental setup
  6. Results and discussion
  7. Uncovering Dataset Bias(es)
  8. Conclusion
  9. Acknowledgements

Figures (4)

  • Figure 1: Identified personal data concepts by PrivLEX for VISPR Orekondy_2017 (a-d) and PrivacyAlert Zhao_2022_privacyalert (e-g) images. The colour in the bottom-right corner of an image indicates the ground-truth label (private/public), and the number corresponds to the predicted probability of the private class. The colour of a concept indicates its contribution to the private or public class, or signals zero weight.
  • Figure 2: Balanced accuracy of CLIP zero-shot recognition of VISPR concepts Orekondy_2017 with two types of concept descriptions: 1) detailed VISPR descriptions provided in the dataset for annotators (VISPR-VI) $\bullet$, 2) legal-style descriptions (VISPR-LS) $\bullet$.
  • Figure 3: Misclassified by PrivLEX images from VISPR Orekondy_2017 (a-d) and PrivacyAlert Zhao_2022_privacyalert (e-g) with identified personal data concepts. The colour in the bottom-right corner of an image indicates the ground-truth label (private/public), and the number corresponds to the predicted probability of the private class. The colour of a concept indicates its contribution to the private or public class, or signals zero weight.
  • Figure 4: Contribution of legal concepts to the private class (PrivacyAlert$\bullet$, VISPR$\bullet$).