Towards Building Non-Fine-Tunable Foundation Models

TL;DR

This work tackles the challenge of open-sourcing foundation models by introducing Private Mask Pre-Training (PMP), a pre-training paradigm that concentrates learning into a private sparse subnetwork defined by a binary mask $M$ with sparsity $\\rho$. The mask is discovered via an Early-Bird lottery ticket, kept private, and used to restrict updates to $\\theta_M$ during pre-training; only the final dense weights are released, forcing unauthorized fine-tuning to operate in a geometry misaligned with pre-training. The authors provide a theoretical argument showing an intrinsic optimization mismatch that destabilizes unauthorized updates, and they validate PMP empirically on LLM-scale models showing preserved base performance and degraded unauthorized fine-tuning across tasks, with a tunable non-fine-tunability via the mask ratio. Authorized fine-tuning with access to the mask improves adaptation relative to unauthorized tuning, while maintaining practical utility. Overall, PMP reframes non-fine-tunability as a pre-training-level property that enables safer open deployment of foundation models without requiring knowledge of downstream data.

Abstract

Open-sourcing foundation models (FMs) enables broad reuse but also exposes model trainers to economic and safety risks from unrestricted downstream fine-tuning. We address this problem by building non-fine-tunable foundation models: models that remain broadly usable in their released form while yielding limited adaptation gains under task-agnostic unauthorized fine-tuning. We propose Private Mask Pre-Training (PMP), a pre-training framework that concentrates representation learning into a sparse subnetwork identified early in training. The binary mask defining this subnetwork is kept private, and only the final dense weights are released. This forces unauthorized fine-tuning without access to the mask to update parameters misaligned with pretraining subspace, inducing an intrinsic mismatch between the fine-tuning objective and the pre-training geometry. We provide theoretical analysis showing that this mismatch destabilizes gradient-based adaptation and bounds fine-tuning gains. Empirical results on large language models demonstrating that PMP preserves base model performance while consistently degrading unauthorized fine-tuning across a wide range of downstream tasks, with the strength of non-fine-tunability controlled by the mask ratio.

Figures (6)

• Figure 1: Overview of Private Mask Pretraining (PMP). Top: the foundation model trainer discovers an Early-Bird lottery ticket and performs pretraining by restricting updates to a private mask. Bottom: after model release, authorized fine-tuning with access to the mask updates only the pretrained ticket and enables effective adaptation, while unauthorized fine-tuning without the mask perturbs both trained and untrained parameters, leading to degraded performance.
• Figure 2: Effect of EarlyBird mask selection on TinyLlama in PMP. We compare PMP with EarlyBird and PMP with random mask selection on (a) base performance and (b) fine-tuned performance.
• Figure 3: Authorized vs. unauthorized fine-tuning on GLUE (TinyLlama). Authorized fine-tuning (with access to the private mask) updates only the masked subspace, while unauthorized fine-tuning performs standard full-parameter updates without the mask. Authorized fine-tuning consistently outperforms unauthorized fine-tuning.
• Figure 4: Impact of Mask Ratio and Optimization Hyperparameters on Fine-Tuning Performance. The left panel shows QNLI performance under different mask ratios, where standard fine-tuning yields substantial gains while PMP consistently suppresses adaptation with minimal impact on base performance. The middle panel evaluates sensitivity to the fine-tuning learning rate, and the right panel reports performance across fine-tuning epochs. Across all settings, PMP consistently limits downstream adaptation, and increasing optimization strength does not recover performance.
• Figure 5: 1D Loss Landscape Visualization. We compare the loss curve along an authorized direction (confined to the mask, Blue) versus an unauthorized direction (full parameter space, Red).