Understanding NLP neural networks by the texts they generate

Abstract

Recurrent neural networks have proven useful in natural language processing. For example, they can be trained to predict, and even generate plausible text with few or no spelling and syntax errors. However, it is not clear what grammar a network has learned, or how it keeps track of the syntactic structure of its input. In this paper, we present a new method to extract a finite state machine from a recurrent neural network. A FSM is in principle a more interpretable representation of a grammar than a neural net would be, however the extracted FSMs for realistic neural networks will also be large. Therefore, we also look at ways to group the states and paths through the extracted FSM so as to get a smaller, easier to understand model of the neural network. To illustrate our methods, we use them to investigate how a neural network learns noun-verb agreement from a simple grammar where relative clauses may appear between noun and verb.