KPT: Keyword-Guided Pre-training for Grounded Dialog Generation

Abstract

Incorporating external knowledge into the response genera tion process is essential to building more helpful and reli able dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we pro pose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog genera tion without relying on extra knowledge annotation. Specifi cally, we use a pre-trained language model to extract the most uncertain tokens in the dialog as keywords. With these key words, we construct two kinds of knowledge and pre-train a knowledge-grounded response generation model, aiming at handling two different scenarios: (1) the knowledge should be faithfully grounded; (2) it can be selectively used. For the former, the grounding knowledge consists of keywords ex tracted from the response. For the latter, the grounding knowl edge is additionally augmented with keywords extracted from other utterances in the same dialog. Since the knowledge is extracted from the dialog itself, KPT can be easily performed on a large volume and variety of dialogue data. We considered three data sources (open-domain, task-oriented, conversa tional QA) with a total of 2.5M dialogues. We conduct exten sive experiments on various few-shot knowledge-grounded generation tasks, including grounding on dialog acts, knowl edge graphs, persona descriptions, and Wikipedia passages. Our comprehensive experiments and analyses demonstrate that KPT consistently outperforms state-of-the-art methods on these tasks with diverse grounding knowledge.