PRISP: Privacy-Safe Few-Shot Personalization via Lightweight Adaptation

TL;DR

PRISP tackles practical LLM personalization under strict data and privacy constraints by generating a task-aware anchor LoRA from natural language descriptions using a Text-to-LoRA hypernetwork and then personalizing with a lightweight bridge on top of a frozen anchor. This two-stage design eliminates the need for task data and cross-user parameter sharing, offering strong few-shot performance with high data efficiency and low computation. The approach achieves competitive results across diverse LaMP tasks, reduces training time, and preserves privacy, making it well suited for edge deployment. Its robustness to task description variations and ability to generalize to unseen tasks demonstrate its broad potential for real-world personalization without compromising user privacy.

Abstract

Large language model (LLM) personalization aims to adapt general-purpose models to individual users. Most existing methods, however, are developed under data-rich and resource-abundant settings, often incurring privacy risks. In contrast, realistic personalization typically occurs after deployment under (i) extremely limited user data, (ii) constrained computational resources, and (iii) strict privacy requirements. We propose PRISP, a lightweight and privacy-safe personalization framework tailored to these constraints. PRISP leverages a Text-to-LoRA hypernetwork to generate task-aware LoRA parameters from task descriptions, and enables efficient user personalization by optimizing a small subset of task-aware LoRA parameters together with minimal additional modules using few-shot user data. Experiments on a few-shot variant of the LaMP benchmark demonstrate that PRISP achieves strong overall performance compared to prior approaches, while reducing computational overhead and eliminating privacy risks.

Figures (9)

• Figure 1: When personalization relies on less than 1% of the full training data, existing methods exhibit substantial performance degradation, highlighting the challenge of robust few-shot personalization. The exact few-shot setting is described later in Table \ref{['tab:fewshot-scenario']}.
• Figure 2: Overview of PRISP in comparison to previous works. Top: Prior methods rely on task data and other users’ parameters (sharer LoRAs) for personalization. Bottom: PRISP generates a task-aware anchor LoRA from a task description without task data via a hypernetwork, and personalizes the model by training only a lightweight bridge module and the $B$ matrix. This enables task-data-free, privacy-safe, and fast personalization.
• Figure 3: Task-averaged peak GPU memory usage and training time on the LaMP benchmark, where values are averaged across all tasks. Our method achieves competitive memory consumption while reducing training time by orders of magnitude compared to prior personalization approaches.
• Figure 4: Cost-performance trade-off on the LaMP benchmark under a few-shot setting. Composite cost is computed by max-normalizing GPU memory usage and training time, and averaging them with equal weights.
• Figure 5: Comparison of anchor LoRAs used for personalization. We compare three anchor LoRAs: (i) a sharer LoRA selected most frequently during PriME optimization, (ii) a task-adapted LoRA trained on 50 task-specific samples, and (iii) our task-aware anchor LoRA generated via a hypernetwork. The reported values denote average performance computed by aggregating the task-specific evaluation metrics for each task.