How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning
Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM.
698d5d9565c0d15a6d162214
Jiahao Yuan
admin_assigned
2026-02-12T16:47:44.410Z
67acb56faee29e2e5c1c41ec
/avatars/b47094266fe70d10786da845ae2ade2c.svg
Jiahao Yuan
Jhcircle
user
Jhcircle
698d5d9565c0d15a6d162215
Yike Xu
698d5d9565c0d15a6d162216
Jinyong Wen
698d5d9565c0d15a6d162217
Baokun Wang
698d5d9565c0d15a6d162218
Yang Chen
698d5d9565c0d15a6d162219
Xiaotong Lin
698d5d9565c0d15a6d16221a
Wuliang Huang
698d5d9565c0d15a6d16221b
Ziyi Gao
698d5d9565c0d15a6d16221c
Xing Fu
698d5d9565c0d15a6d16221d
Yu Cheng
698d5d9565c0d15a6d16221e
Weiqiang Wang
How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning
Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM.
Research investigates the impact of different attention masking strategies on user embedding quality in decoder-only language models, proposing a gradient-guided soft masking technique to improve training stability and representation quality for user behavior analysis.
decoder-only large language models attention masking causal attention hybrid attention bidirectional attention contrastive learning gradient-guided soft masking pre-warmup linear scheduler user representation learning user cognition benchmarks