HuRef: HUman-REadable Fingerprint for Large Language Models

TL;DR

HuRef introduces a black-box fingerprinting approach for large language models that identifies the base model by leveraging a stable parameter-direction signal. It derives three invariant terms from the Transformer that are robust to weight rearrangements, maps these invariants to Gaussian vectors, and then to human-readable fingerprint images produced by StyleGAN2, all under zero-knowledge proofs to prevent information leakage. The method demonstrates high discriminative power across base-offspring pairs (ICS ≈ 100% for shared bases) and low similarity across independently trained models, while a human-subject study confirms practical interpretability. This work enables copyright protection and provenance tracking for LLMs without exposing weights or requiring access to internal training data, offering a scalable, privacy-preserving solution for base-model identification. The approach is transformer-specific, scalable, and supported by quantitative results and cryptographic guarantees, with code available at the provided repository.

Abstract

Protecting the copyright of large language models (LLMs) has become crucial due to their resource-intensive training and accompanying carefully designed licenses. However, identifying the original base model of an LLM is challenging due to potential parameter alterations. In this study, we introduce HuRef, a human-readable fingerprint for LLMs that uniquely identifies the base model without interfering with training or exposing model parameters to the public. We first observe that the vector direction of LLM parameters remains stable after the model has converged during pretraining, with negligible perturbations through subsequent training steps, including continued pretraining, supervised fine-tuning, and RLHF, which makes it a sufficient condition to identify the base model. The necessity is validated by continuing to train an LLM with an extra term to drive away the model parameters' direction and the model becomes damaged. However, this direction is vulnerable to simple attacks like dimension permutation or matrix rotation, which significantly change it without affecting performance. To address this, leveraging the Transformer structure, we systematically analyze potential attacks and define three invariant terms that identify an LLM's base model. Due to the potential risk of information leakage, we cannot publish invariant terms directly. Instead, we map them to a Gaussian vector using an encoder, then convert it into a natural image using StyleGAN2, and finally publish the image. In our black-box setting, all fingerprinting steps are internally conducted by the LLMs owners. To ensure the published fingerprints are honestly generated, we introduced Zero-Knowledge Proof (ZKP). Experimental results across various LLMs demonstrate the effectiveness of our method. The code is available at https://github.com/LUMIA-Group/HuRef.

Figures (12)

• Figure 1: An illustrative framework for LLM protection with fingerprints. The LLM manufacturers compute invariant terms internally and feed them into the fingerprinting model (FPM) to generate a fingerprint image. This image is then released to the public along with zero-knowledge proofs ($\pi_1$), allowing for intuitive identification of shared base models through the fingerprint images. We also provide a limited one-to-one quantitative comparison scheme (ICS & $\pi_2$) as a complement. Zero-Knowledge Proof guarantees the reliability of the fingerprints and comparison results, without interfering with LLM training or revealing model parameters to the public.
• Figure 2: The model's performance quickly deteriorates as the PCS decreases.
• Figure 3: Transformer layer
• Figure 4: The training and inference of our fingerprinting model.
• Figure 5: Fingerprints of 7 different base models (in the first row) and their corresponding offspring models (the lower two rows) are presented. The base model's name is omitted in the offspring models.