Robust Prompt Optimization for Large Language Models Against Distribution Shifts

1Citations
Citations of this article
19Readers
Mendeley users who have this article in their library.

Abstract

Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. However, their effectiveness is highly dependent on the phrasing of the task prompt, leading to research on automatic prompt optimization using labeled task data. We reveal that these prompt optimization techniques are vulnerable to distribution shifts such as subpopulation shifts, which are common for LLMs in real-world scenarios such as customer reviews analysis. In this light, we propose a new problem of robust prompt optimization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group. To solve this problem, we propose Generalized Prompt Optimization framework, which incorporates the unlabeled data from the target group into prompt optimization. Extensive experimental results demonstrate the effectiveness of the proposed framework with significant performance improvement on the target group and comparable performance on the source group.

Cite

CITATION STYLE

APA

Li, M., Wang, W., Feng, F., Cao, Y., Zhang, J., & Chua, T. S. (2023). Robust Prompt Optimization for Large Language Models Against Distribution Shifts. In EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 1539–1554). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.emnlp-main.95

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free