Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy

Stephen Meisenbacher, Maulik Chevli, Florian Matthes

TL;DR

This work tackles semantic coherence and output length limitations of word-level Metric Differential Privacy (MDP) in NLP by introducing collocation-based perturbations that operate on 1–3 token n-grams. It builds a joint embedding space for unigrams, bigrams, and trigrams and trains two collocation extraction/tokenization schemes (GST and MST). By augmenting an existing MLDP mechanism (MADLIB) to perturb collocation tokens, the method achieves higher utility and more coherent outputs, with comparable privacy, across GLUE tasks and targeted privacy evaluations. The study demonstrates the practicality of collocation-level privatization and highlights avenues for expanding DP-NLP beyond word-level tokenization, while acknowledging PMI-based extraction limitations and the need for broader mechanism validation.

Abstract

Applications of Differential Privacy (DP) in NLP must distinguish between the syntactic level on which a proposed mechanism operates, often taking the form of $\textit{word-level}$ or $\textit{document-level}$ privatization. Recently, several word-level $\textit{Metric}$ Differential Privacy approaches have been proposed, which rely on this generalized DP notion for operating in word embedding spaces. These approaches, however, often fail to produce semantically coherent textual outputs, and their application at the sentence- or document-level is only possible by a basic composition of word perturbations. In this work, we strive to address these challenges by operating $\textit{between}$ the word and sentence levels, namely with $\textit{collocations}$. By perturbing n-grams rather than single words, we devise a method where composed privatized outputs have higher semantic coherence and variable length. This is accomplished by constructing an embedding model based on frequently occurring word groups, in which unigram words co-exist with bi- and trigram collocations. We evaluate our method in utility and privacy tests, which make a clear case for tokenization strategies beyond the word level.

A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy

TL;DR

This work tackles semantic coherence and output length limitations of word-level Metric Differential Privacy (MDP) in NLP by introducing collocation-based perturbations that operate on 1–3 token n-grams. It builds a joint embedding space for unigrams, bigrams, and trigrams and trains two collocation extraction/tokenization schemes (GST and MST). By augmenting an existing MLDP mechanism (MADLIB) to perturb collocation tokens, the method achieves higher utility and more coherent outputs, with comparable privacy, across GLUE tasks and targeted privacy evaluations. The study demonstrates the practicality of collocation-level privatization and highlights avenues for expanding DP-NLP beyond word-level tokenization, while acknowledging PMI-based extraction limitations and the need for broader mechanism validation.

Abstract

Applications of Differential Privacy (DP) in NLP must distinguish between the syntactic level on which a proposed mechanism operates, often taking the form of or privatization. Recently, several word-level Differential Privacy approaches have been proposed, which rely on this generalized DP notion for operating in word embedding spaces. These approaches, however, often fail to produce semantically coherent textual outputs, and their application at the sentence- or document-level is only possible by a basic composition of word perturbations. In this work, we strive to address these challenges by operating the word and sentence levels, namely with . By perturbing n-grams rather than single words, we devise a method where composed privatized outputs have higher semantic coherence and variable length. This is accomplished by constructing an embedding model based on frequently occurring word groups, in which unigram words co-exist with bi- and trigram collocations. We evaluate our method in utility and privacy tests, which make a clear case for tokenization strategies beyond the word level.
Paper Structure (37 sections, 8 equations, 2 figures, 9 tables, 3 algorithms)

This paper contains 37 sections, 8 equations, 2 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Foundations
  3. Differential Privacy
  4. Metric Differential Privacy (MDP)
  5. MDP for a Sentence
  6. The Theory of Collocations
  7. Related Work
  8. Word-level MLDP
  9. Collocation Extraction and Evaluation
  10. N-gram Embeddings
  11. A Collocation-based MLDP Method
  12. Extracting Collocations
  13. Empirical Collocations
  14. Collocation-level Tokenization
  15. A Collocation Embedding Space
  16. ...and 22 more sections

Figures (2)

  • Figure 1: An example of word tokenization versus collocation tokenization. Collocation tokenization will often result in fewer tokens, as collocations frequently occur in natural language. Token budget denotes the privacy budget assigned to each token given an example document-level budget (e.g., $\varepsilon = 10$) and assuming basic composition.
  • Figure 2: Average Utility Loss. This graph depicts the average utility loss (in F1) for a given base $\varepsilon$ value across four GLUE tasks and our four privatization strategies.