Taylor Unswift: Secured Weight Release for Large Language Models via Taylor Expansion

TL;DR

TaylorMLP addresses the conflict between sharing LLMs (ownership and data privacy) and enabling safe access by transforming MLP weights into Taylor-series parameters, thus securing the weights and preventing reconstruction. By estimating a local embedding $\mathbf{z}_0$ and representing the MLP forward pass as $y_i=\sum_{n=0}^{N} \langle \boldsymbol{\Theta}_{i,n}, (\mathbf{z}-\mathbf{z}_0)^n \rangle$, TaylorMLP allows token generation without exposing original weights, and increases latency roughly by a factor of $N$ to deter unauthorized use (Taylor Unswift). Theoretical convergence shows outputs approach those of the original MLP as $N$ grows, with practical sufficiency at $N\ge 8$, while experiments across five datasets and three LLMs demonstrate near-original accuracy alongside substantial latency increases and defense against fine-tuning and distillation. The approach enables secure sharing and testing under regulatory constraints, with complementary watermarking for misuse detection. Overall, TaylorMLP offers a practical, model-agnostic mechanism to protect ownership and curb abuse of released LLMs without sacrificing core capabilities.

Abstract

Ensuring the security of released large language models (LLMs) poses a significant dilemma, as existing mechanisms either compromise ownership rights or raise data privacy concerns. To address this dilemma, we introduce TaylorMLP to protect the ownership of released LLMs and prevent their abuse. Specifically, TaylorMLP preserves the ownership of LLMs by transforming the weights of LLMs into parameters of Taylor-series. Instead of releasing the original weights, developers can release the Taylor-series parameters with users, thereby ensuring the security of LLMs. Moreover, TaylorMLP can prevent abuse of LLMs by adjusting the generation speed. It can induce low-speed token generation for the protected LLMs by increasing the terms in the Taylor-series. This intentional delay helps LLM developers prevent potential large-scale unauthorized uses of their models. Empirical experiments across five datasets and three LLM architectures demonstrate that TaylorMLP induces over 4x increase in latency, producing the tokens precisely matched with original LLMs. Subsequent defensive experiments further confirm that TaylorMLP effectively prevents users from reconstructing the weight values based on downstream datasets.

Figures (7)

• Figure 1: Existing mechanisms for releasing LLMs: (a) API release and (b) open-source. We propose (c) TaylorMLP to protect the ownership of released LLMs.
• Figure 2: (a) Original MLP layers parameterized by $\textbf{V}, \textbf{b}, \textbf{W}$, and $\textbf{c}$. (b) TaylorMLP layers. TaylorMLP converts $\textbf{b}, \textbf{W}_i$ and $c_i$ into $\{\boldsymbol{\Theta}_{i,0}, \cdots, \boldsymbol{\Theta}_{i,N}\}$ for securing their values. This process is irreversible.
• Figure 3: The outputs of original LLMs, distilled LLMs, and TaylorMLP. The input context is from the wikitext-2 dataset.
• Figure 4: The input context and question from the CoQA dataset; the outputs of original LLMs; and the outputs of TaylorMLP whose the expansion order $N$ takes 0, 1, 2, and 8.
• Figure 5: The KL divergence of output probability and ROUGE-1 score versus the expansion order.