Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized LoRA for Human-Centered Text Understanding

You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, Xuejie Zhang

TL;DR

This work presents PLoRA, a personalized low-rank adaptation that fuses Personalized Knowledge Injection with LoRA to personalize PLMs for human-centered text understanding while remaining parameter-efficient. A Plug-and-Play framework, incorporating PDropout and Mutual Information Maximization, enables robust zero-shot and few-shot learning for cold-start users without incurring additional inference latency. Empirical results on IMDB, YELP, GDRD, and PPR show that PLoRA improves full-, few-, and zero-shot performance with fewer trainable parameters than traditional fine-tuning approaches, across BERT, RoBERTa, and Flan-T5 backbones. The method’s flexibility and efficiency suggest broad applicability to other PLMs and tasks beyond sentiment analysis, with code released for reproducibility.

Abstract

Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.

Personalized LoRA for Human-Centered Text Understanding

TL;DR

This work presents PLoRA, a personalized low-rank adaptation that fuses Personalized Knowledge Injection with LoRA to personalize PLMs for human-centered text understanding while remaining parameter-efficient. A Plug-and-Play framework, incorporating PDropout and Mutual Information Maximization, enables robust zero-shot and few-shot learning for cold-start users without incurring additional inference latency. Empirical results on IMDB, YELP, GDRD, and PPR show that PLoRA improves full-, few-, and zero-shot performance with fewer trainable parameters than traditional fine-tuning approaches, across BERT, RoBERTa, and Flan-T5 backbones. The method’s flexibility and efficiency suggest broad applicability to other PLMs and tasks beyond sentiment analysis, with code released for reproducibility.

Abstract

Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.
Paper Structure (37 sections, 8 equations, 7 figures, 6 tables)

This paper contains 37 sections, 8 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Personalized LoRA
  5. Task-specific LoRA
  6. Personalization-specific PKI
  7. PLoRA
  8. Plug-and-Play for Cold Start
  9. Zero-shot learning scenarios
  10. Few-shot learning scenarios
  11. Train and Inference
  12. Experiments
  13. Datasets and Evaluation
  14. Datasets
  15. Evaluation
  16. ...and 22 more sections

Figures (7)

  • Figure 1: Different methods for human-centered text understanding.
  • Figure 2: An illustration of PLoRA.
  • Figure 3: A diagram of MIM on PLoRA for zero-shot learn-ing.
  • Figure 4: A diagram of PnP framework on few-shot learning scenarios for user D (UD).
  • Figure 5: Effect of PLoRA with different data sparsity. FuS, FS, and ZS in figures (including the following figures) means using full/few/zero-shot learning methods, respectively. {$\cal M$}-PLoRA {A/B}-{FuS/ZS/FS} corresponds dev figures of PLM $\cal M$-based PLoRA applied for datasets (A or B) with FuS/ZS/FS methods. $\approx 100\%$ means almost the full training dataset in ${{{\cal D}}^{\rm{B}}}$ is used and also presents no data sparsity for every user in ${{{\cal D}}^{\rm{B}}}$.
  • ...and 2 more figures