How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning

TL;DR

This work analyzes how attention masking in decoder-only LLMs shapes user embeddings, proposing Gradient-Guided Soft Masking (GG-SM) to stabilize the transition from causal to bidirectional attention within a unified contrastive framework. It builds two synthesis-enabled data streams—Rule-based Behavioral Trajectories and LLM-synthesized QA alignments—and a calibration-and-difficulty probing pipeline to curate challenging positives for robust learning, underpinned by the InfoNCE objective $L_{cl}$. On nine industrial benchmarks using real-world Alipay data, GG-SM yields more stable training and higher-quality bidirectional representations than causal, hybrid, or scheduler baselines, with strong domain robustness and transferability, while remaining compatible with decoder pretraining. The findings provide practical guidance for masking design and training transitions in deploying decoder-only LLMs as effective user encoders, and the authors share their code to support reproducibility and adoption. These insights advance scalable, long-horizon user representation learning in real-world, multimodal settings.

Abstract

Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM.

Figures (4)

• Figure 1: Architechure Overview of Our Find-Embedding (w / GGSM).
• Figure 2: Average AUC performance across 9 downstream tasks under different attention masking strategies (left) and comparison with general embedding, user embedding (right).
• Figure 3: Training loss convergence: GG-SM (Ours) vs. scheduler.
• Figure 4: Input format of Training Data. <USER> token serves as the anchor for extracting user embedding.