Robust User Behavioral Sequence Representation via Multi-scale Stochastic Distribution Prediction

Abstract

User behavior representation learned by self-supervised pre-training tasks is widely used in various domains and applications. Conventional methods usually follow the methodology in Natural Language Processing (NLP) to set the pre-training tasks. They either randomly mask some of the behaviors in the sequence and predict the masked ones or predict the next k behaviors. These methods fit for text sequence, in which the tokens are sequentially arranged subject to linguistic criterion. However, the user behavior sequences can be stochastic with noise and randomness. The same paradigm is intractable for learning a robust user behavioral representation. Though the next user behavior can be stochastic, the behavior distribution over a period of time is much more stable and less noisy. Based on this, we propose a Multi-scale Stochastic Distribution Prediction (MSDP) algorithm for learning robust user behavioral sequence representation. Instead of using predictions on concrete behavior as pre-training tasks, we take the prediction on user's behaviors distribution over a period of time as the self-supervision signal. Moreover, inspired by the recent success of the multi-task prompt training method on Large Language Models (LLM), we propose using the window size of the predicted time period as a prompt, enabling the model to learn user behavior representations that can be applied to prediction tasks across various future time periods. We generate different window size prompts through stochastic sampling. It effectively improves the generalization capability of the learned sequence representation. Extensive experiments demonstrate that our approach can learn robust user behavior representation successfully, which significantly outperforms state-of-the-art (SOTA) baselines.