Step-Back Profiling: Distilling User History for Personalized Scientific Writing

TL;DR

This work tackles personalized content generation with large language models by introducing Step-back Profiling, which distills each user's history into compact gist-based profiles for efficient, multi-user personalization. The approach is training-free and supports single- and multi-author collaboration, with optional retrieval augmentation to incorporate history snippets. The authors also contribute the Personalized Scientific Writing (PSW) benchmark to simulate team-based scientific writing and evaluate profiling across UP-0 and PSW tasks, combining GPT-based evaluation. Empirical results on LaMP and PSW show consistent improvements over non-personalized and retrieval-based baselines, with multi-user settings delivering the strongest gains and ablations highlighting the importance of preserved author order and faithful user profiling. The work advances practical, scalable personalization for collaborative writing and provides a foundation for future enhancements in dynamic profiling, privacy-preserving methods, and broader domain testing.

Abstract

Large language models (LLM) excel at a variety of natural language processing tasks, yet they struggle to generate personalized content for individuals, particularly in real-world scenarios like scientific writing. Addressing this challenge, we introduce STEP-BACK PROFILING to personalize LLMs by distilling user history into concise profiles, including essential traits and preferences of users. To conduct the experiments, we construct a Personalized Scientific Writing (PSW) dataset to study multi-user personalization. PSW requires the models to write scientific papers given specialized author groups with diverse academic backgrounds. As for the results, we demonstrate the effectiveness of capturing user characteristics via STEP-BACK PROFILING for collaborative writing. Moreover, our approach outperforms the baselines by up to 3.6 points on the general personalization benchmark (LaMP), including 7 personalization LLM tasks. Our ablation studies validate the contributions of different components in our method and provide insights into our task definition. Our dataset and code are available at \url{https://github.com/gersteinlab/step-back-profiling}.

Figures (5)

• Figure 1: Overview of the Step-back Profiling.Step-back Profiling creates compact profile representations for each user and augments LLM generation with personalized information.
• Figure 2: Step-back Profiling performance on the LaMP benchmark.We observe dominant performance of Step-back Profiling on most LaMP tasks. Details of experimental setup can be found in Section \ref{['sec:main_result']}.
• Figure 3: Performance metrics across three different models: Zero-shot, Single-Author, and Multi-Author. The Multi-Author model consistently achieves the highest scores across all datasets.
• Figure 4: Impact of author order on the performance across three different models: Original, Swap-Random, and Swap-First. The Original model consistently achieves the highest scores across all datasets.
• Figure 5: Impact of user profiling on the performance across three different models:Original, Removed, and Random. The Original model consistently achieves the highest scores across all datasets.