Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REEF: Representation Encoding Fingerprints for Large Language Models

Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, Jing Shao

TL;DR

A training-free REEF is proposed to identify the relationship between the suspect and victim models from the perspective of LLMs' feature representations and computes and compares the centered kernel alignment similarity between the representations of a suspect model and a victim model on the same samples.

Abstract

Protecting the intellectual property of open-source Large Language Models (LLMs) is very important, because training LLMs costs extensive computational resources and data. Therefore, model owners and third parties need to identify whether a suspect model is a subsequent development of the victim model. To this end, we propose a training-free REEF to identify the relationship between the suspect and victim models from the perspective of LLMs' feature representations. Specifically, REEF computes and compares the centered kernel alignment similarity between the representations of a suspect model and a victim model on the same samples. This training-free REEF does not impair the model's general capabilities and is robust to sequential fine-tuning, pruning, model merging, and permutations. In this way, REEF provides a simple and effective way for third parties and models' owners to protect LLMs' intellectual property together. The code is available at https://github.com/tmylla/REEF.

REEF: Representation Encoding Fingerprints for Large Language Models

TL;DR

A training-free REEF is proposed to identify the relationship between the suspect and victim models from the perspective of LLMs' feature representations and computes and compares the centered kernel alignment similarity between the representations of a suspect model and a victim model on the same samples.

Abstract

Protecting the intellectual property of open-source Large Language Models (LLMs) is very important, because training LLMs costs extensive computational resources and data. Therefore, model owners and third parties need to identify whether a suspect model is a subsequent development of the victim model. To this end, we propose a training-free REEF to identify the relationship between the suspect and victim models from the perspective of LLMs' feature representations. Specifically, REEF computes and compares the centered kernel alignment similarity between the representations of a suspect model and a victim model on the same samples. This training-free REEF does not impair the model's general capabilities and is robust to sequential fine-tuning, pruning, model merging, and permutations. In this way, REEF provides a simple and effective way for third parties and models' owners to protect LLMs' intellectual property together. The code is available at https://github.com/tmylla/REEF.

Paper Structure

This paper contains 29 sections, 2 theorems, 27 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Injected Fingerprint.
  4. Intrinsic Fingerprint.
  5. Exploring the Potential of Feature Representations as Fingerprints
  6. Classifiers trained on representations of a victim model can effectively generalize to its variants but not to others.
  7. Robust Representation-based Fingerprinting with REEF
  8. Centered Kernel Alignment.
  9. Kernel Selection.
  10. Experiments
  11. Effectiveness Verification
  12. Robustness Verification
  13. Baseline Methods
  14. Fine-tuning
  15. Model Pruning
  16. ...and 14 more sections

Key Result

Theorem 1

(Proof in Appendix ap:proof) Given two matrices $X \in \mathbb{R}^{m \times p_1}$ and $Y \in \mathbb{R}^{m \times p_2}$, the CKA similarity score between $X$ and $Y$ is invariant under any permutation of the columns and column-wise scaling transformation. Formally, we have: where $P_1 \in \mathbb{R}^{p_1 \times p_1}$ and $P_2 \in \mathbb{R}^{p_2 \times p_2}$ denote permutation matrices. $c_1 \in

Figures (9)

  • Figure 1: (a) t-SNE visualization of different LLMs’ representations on the same samples. (b) Performance of classifiers trained on representations from the victim model evaluated on suspect models. (c) Robustness of REEF under variant LLMs that cause ICS zeng2023huref ineffective.
  • Figure 2: Accuracies of classifiers trained on representations from the victim model: (a) Llama-2-7b as the victim model, (b) Llama-2-13b as the victim model.
  • Figure 3: Heatmaps depicting the CKA similarity between the representations of the victim LLM (Llama-2-7B) and those of various suspect LLMs on the same samples.
  • Figure 4: (a)-(c) Similarity between pruned models and the victim model across three pruning strategies at various pruning ratios. (d) Perplexity of the three pruning strategies.
  • Figure 5: Illustration of the CKA similarity between the representations of the victim LLM (Llama-2-7B) and various suspect LLMs across different datasets as sample number increases.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 1