Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction

Jiahui Gong, Jingtao Ding, Fanjin Meng, Guilong Chen, Hong Chen, Shen Zhao, Haisheng Lu, Yong Li

TL;DR

The paper tackles on-device user intent prediction by adapting pretrained language models to capture both population-level patterns and individual long-tail preferences. It introduces PITuning, a two-stage framework that first performs population-level tuning with event-to-intent transition modeling and event-reconstruction, then distills a lightweight predictor and applies adaptive unlearning before personalized finetuning on device data. The approach yields substantial gains over strong baselines on real-world Honor and Mobile datasets, notably narrowing the gap for long-tail intents and achieving up to ~37% macro-precision/recall improvements and up to ~11% NDCG gains. The framework demonstrates practicality for mobile deployment through efficient distillation and an effective bias-correction strategy, enabling accurate, private, on-device user intent prediction. This work advances cross-domain PLM adaptation for behavioral tasks and offers a scalable path to personalized mobile assistants.

Abstract

Mobile devices, especially smartphones, can support rich functions and have developed into indispensable tools in daily life. With the rise of generative AI services, smartphones can potentially transform into personalized assistants, anticipating user needs and scheduling services accordingly. Predicting user intents on smartphones, and reflecting anticipated activities based on past interactions and context, remains a pivotal step towards this vision. Existing research predominantly focuses on specific domains, neglecting the challenge of modeling diverse event sequences across dynamic contexts. Leveraging pre-trained language models (PLMs) offers a promising avenue, yet adapting PLMs to on-device user intent prediction presents significant challenges. To address these challenges, we propose PITuning, a Population-to-Individual Tuning framework. PITuning enhances common pattern extraction through dynamic event-to-intent transition modeling and addresses long-tailed preferences via adaptive unlearning strategies. Experimental results on real-world datasets demonstrate PITuning's superior intent prediction performance, highlighting its ability to capture long-tailed preferences and its practicality for on-device prediction scenarios.

A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction

TL;DR

The paper tackles on-device user intent prediction by adapting pretrained language models to capture both population-level patterns and individual long-tail preferences. It introduces PITuning, a two-stage framework that first performs population-level tuning with event-to-intent transition modeling and event-reconstruction, then distills a lightweight predictor and applies adaptive unlearning before personalized finetuning on device data. The approach yields substantial gains over strong baselines on real-world Honor and Mobile datasets, notably narrowing the gap for long-tail intents and achieving up to ~37% macro-precision/recall improvements and up to ~11% NDCG gains. The framework demonstrates practicality for mobile deployment through efficient distillation and an effective bias-correction strategy, enabling accurate, private, on-device user intent prediction. This work advances cross-domain PLM adaptation for behavioral tasks and offers a scalable path to personalized mobile assistants.

Abstract

Mobile devices, especially smartphones, can support rich functions and have developed into indispensable tools in daily life. With the rise of generative AI services, smartphones can potentially transform into personalized assistants, anticipating user needs and scheduling services accordingly. Predicting user intents on smartphones, and reflecting anticipated activities based on past interactions and context, remains a pivotal step towards this vision. Existing research predominantly focuses on specific domains, neglecting the challenge of modeling diverse event sequences across dynamic contexts. Leveraging pre-trained language models (PLMs) offers a promising avenue, yet adapting PLMs to on-device user intent prediction presents significant challenges. To address these challenges, we propose PITuning, a Population-to-Individual Tuning framework. PITuning enhances common pattern extraction through dynamic event-to-intent transition modeling and addresses long-tailed preferences via adaptive unlearning strategies. Experimental results on real-world datasets demonstrate PITuning's superior intent prediction performance, highlighting its ability to capture long-tailed preferences and its practicality for on-device prediction scenarios.
Paper Structure (36 sections, 15 equations, 11 figures, 5 tables, 1 algorithm)

This paper contains 36 sections, 15 equations, 11 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Data Analysis
  4. Problem Statement
  5. method
  6. Framework Overview
  7. Population-level Tuning
  8. Embedding layer.
  9. Transformer block.
  10. Intent-aware attentive modeling.
  11. Event-reconstruction auxiliary loss.
  12. Model distilling.
  13. Individual-level Tuning
  14. Adaptive unlearning on biased intents.
  15. Finetuning for personalized model.
  16. ...and 21 more sections

Figures (11)

  • Figure 1: Distribution gap exists between population-level preference and individual-level preference (by comparing frequency histogram of user intent).
  • Figure 2: The workflow of PITuning framework.
  • Figure 3: (a) The intent predictor architecture. (b) The masked event reconstruction in the population-level tuning. (c) The adaptive unlearning in the individual-level tuning.
  • Figure 4: Ablation study.
  • Figure 5: Comparing performance without pretrained LM.
  • ...and 6 more figures