Regeneration Learning: A Learning Paradigm for Data Generation

Abstract

Machine learning methods for conditional data generation usually build a mapping from source conditional data X to target data Y. The target Y (e.g., text, speech, music, image, video) is usually high-dimensional and complex, and contains information that does not exist in source data, which hinders effective and efficient learning on the source-target mapping. In this paper, we present a learning paradigm called regeneration learning for data generation, which first generates Y′ (an abstraction/representation of Y ) from X and then generates Y from Y′. During training, Y′ is obtained from Y through either handcrafted rules or self-supervised learning and is used to learn X → Y′ and Y′ → Y . Regeneration learning extends the concept of representation learning to data generation tasks, and can be regarded as a counterpart of traditional representation learning, since 1) regeneration learning handles the abstraction (Y′) of the target data Y for data generation while traditional representation learning handles the abstraction (X′) of source data X for data understanding; 2) both the processes of Y′ → Y in regeneration learning and X → X′ in representation learning can be learned in a self-supervised way (e.g., pre-training); 3) both the mappings from X to Y′ in regeneration learning and from X′ to Y in representation learning are simpler than the direct mapping from X to Y . We show that regeneration learning can be a widely-used paradigm for data generation (e.g., text generation, speech recognition, speech synthesis, music composition, image generation, and video generation) and can provide valuable insights into developing data generation methods.