Learning a Better Initialization for Soft Prompts via Meta-Learning

Abstract

Prompt tuning (PT) is an effective approach to adapting pre-trained language models to downstream tasks. However, prompt tuning doesn’t perform well under few-shot settings due to the poor initialization. So pre-trained prompt tuning (PPT) (Gu et al., 2022) is proposed to adapt prompt tuning to few-shot settings by initializing prompts with source data. We propose Meta-learned Prompt Tuning (MetaPT) to further improve PPT’s few-shot learning performance by considering latent structure within the source data. Specifically, we introduce the framework by first clustering source data into different meta-training tasks in an unsupervised manner. Then we leverage these tasks to meta-train prompts with a meta-learning algorithm. Such a process enables prompts to learn a better initialization by discovering commonalities among these meta-training tasks. We evaluate our method on seven downstream sentiment tasks. The results demonstrate that our MetaPT achieves better performance and stability than the state-of-the-art method.