What Can a Generative Language Model Answer About a Passage?

Abstract

Generative language models trained on large, diverse corpora can answer questions about a passage by generating the most likely continuation of the passage in which the answer to a given question, as appended to the passage, is the most likely continuation of that passage. However, accuracy rates vary depending on the type of question asked. In this paper, we keep the passage fixed, and test with a wide variety of question types, exploring the strengths and weaknesses of the GPT-3 language model. We provide the passage and test questions as a challenge set for other language models.