Weakly Supervised Training of Semantic Parsers Jayant Krishnamurthy Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 jayantk@cs.cmu.edu Tom M. Mitchell Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 tom.mitchell@cmu.edu Abstract We present a method for training a semantic parser using only a knowledge base and an un- labeled text corpus, without any individually annotated sentences. Our key observation is that multiple forms of weak supervision can be combined to train an accurate semantic parser: semantic supervision from a knowledge base, and syntactic supervision from dependency- parsed sentences. We apply our approach to train a semantic parser that uses 77 rela- tions from Freebase in its knowledge repre- sentation. This semantic parser extracts in- stances of binary relations with state-of-the- art accuracy, while simultaneously recovering much richer semantic structures, such as con- junctions of multiple relations with partially shared arguments. We demonstrate recovery of this richer structure by extracting logical forms from natural language queries against Freebase. On this task, the trained semantic parser achieves 80% precision and 56% recall, despite never having seen an annotated logical form. 1 Introduction Semantic parsing converts natural language state- ments into logical forms in a meaning repre- sentation language. For example, the phrase ���town in California��� might be represented as ��x.CITY(x) ��� LOCATEDIN(x, CALIFORNIA), where CITY, LOCATEDIN and CALIFORNIA are predicates and entities from a knowledge base. The expressiv- ity and utility of semantic parsing is derived from this meaning representation, which is essentially a program that is directly executable by a computer. In this sense, broad coverage semantic parsing is the goal of natural language understanding. Unfortunately, due to data annotation constraints, modern semantic parsers only operate in narrow do- mains. The best performing semantic parsers are trained using extensive manual annotation: typi- cally, a number of sentences must be annotated with their desired logical form. Although other forms of supervision exist (Clarke et al., 2010 Liang et al., 2011), these methods similarly require annotations for individual sentences. More automated training methods are required to produce semantic parsers with richer meaning representations. This paper presents an algorithm for training a se- mantic parser without per-sentence annotations. In- stead, our approach exploits two easily-obtainable sources of supervision: a large knowledge base and (automatically) dependency-parsed sentences. The semantic parser is trained to identify relation in- stances from the knowledge base while simulta- neously producing parses that syntactically agree with the dependency parses. Combining these two sources of supervision allows us to train an accurate semantic parser for any knowledge base without an- notated training data. We demonstrate our approach by training a Com- binatory Categorial Grammar (CCG) (Steedman, 1996) that parses sentences into logical forms con- taining any of 77 relations from Freebase. Our training data consists of relation instances from Freebase and automatically dependency-parsed sen- tences from a web corpus. The trained semantic parser extracts binary relations with state-of-the-art performance, while recovering considerably richer semantic structure. We demonstrate recovery of this semantic structure using natural language queries

town N : ��x.CITY(x) Lex in (N\N)/N : ��f.��g.��x.5y.f(y) ��� g(x) ��� LOCATEDIN(x,y) Lex California N : ��x.x = CALIFORNIA Lex N\N : ��g.��x.5y.y = CALIFORNIA ��� g(x) ��� LOCATEDIN(x,y) N : ��x.5y.y = CALIFORNIA ��� CITY(x) ��� LOCATEDIN(x,y) Figure 1: An example parse of ���town in California��� using the example CCG lexicon. The first stage in parsing retrieves a category from each word from the lexicon, represented by the ���Lex��� entries. The second stage applies CCG combination rules, in this case both forms of function application, to combine these categories into a semantic parse. against Freebase. Our weakly-supervised semantic parser predicts the correct logical form for 56% of queries, despite never seeing a labeled logical form. This paper is structured as follows. We first pro- vide some background information on CCG and the structure of a knowledge base in Section 2. Section 3 formulates the weakly supervised training prob- lem for semantic parsers and presents our algorithm. Section 4 describes how we applied our algorithm to construct a semantic parser for Freebase, and Sec- tion 5 presents our results. We conclude with related work and discussion. 2 Background 2.1 Combinatory Categorial Grammar Combinatory Categorial grammar (CCG) is a lin- guistic formalism that represents both the syntax and semantics of language (Steedman, 1996). CCG is a lexicalized formalism that encodes all grammatical information in a lexicon ��. This lexicon contains syntactic and semantic categories for each word. A lexicon may include entries such as: town := N : ��x.CITY(x) California := N : ��x.x = CALIFORNIA in := (N\N)/N : ��f.��g.��x. 5y.f(y) ��� g(x) ��� LOCATEDIN(x,y) Each entry of the lexicon w := s : l maps a word or short phrase w to a syntactic category s and a logical form l. Syntactic categories s may be atomic (N) or complex (N\N). Logical forms l are lambda calcu- lus expressions constructed using predicates from a knowledge base. These logical forms combine dur- ing parsing to form a complete logical form for the parsed text. Parses are constructed by combining adjacent cat- egories using several combination rules, such as for- ward ( ) and backward ( ) application: X/Y : f Y : g =��� X : f(g) ( ) Y : g X\Y : f =��� X : f(g) ( ) These rules mean that the complex category X/Y (X\Y ) behaves like a function which accepts an ar- gument of type Y on its right (left) and returns a value of type X. Parsing amounts to sequentially applying these two rules, as shown in Figure 1. The result of parsing is an ordered pair, containing both a syntactic parse tree and an associated logical form. We refer to such an ordered pair as a semantic parse, or by using the letter ���. Given a lexicon, there may be multiple seman- tic parses ��� for a given phrase w. Like context-free grammars (CFGs), CCGs can be extended to repre- sent a probability distribution over parses P(���|w ��) where �� is a parameter vector. 2.2 Knowledge Base The main input to our system is a propositional knowledge base K = (E,R,C, ���), containing entities E, categories C, relations R and relation instances ���. Categories and relations are pred- icates which operate on entities and return truth values categories c ��� C are one-place predi- cates (CITY(e)) and relations r ��� R are two- place predicates (LOCATEDIN(e1,e2)). Entities e ��� E represent real-world entities and have a set of known text names. For example, CALIFORNIA is an entity whose text names include ���Califor- nia��� and ���CA.��� Relation instances r(e1,e2) ��� ��� are facts asserted by the knowledge base, such as LOCATEDIN(SACRAMENTO, CALIFORNIA). Ex- amples of such knowledge bases include Freebase (Bollacker et al., 2008), NELL (Carlson et al., 2010), and YAGO (Suchanek et al., 2007). The knowledge base influences the semantic parser in two ways. First, CCG logical forms are constructed by combining categories, relations and entities from the knowledge base with logical con- nectives hence, the predicates in the knowledge base determine the expressivity of the parser���s se- mantic representation. Second, the known relation