Training and scaling preference functions for disambiguation

Abstract

We present an automatic method for weighting the contributions of preference functions used in disambiguation. Initial scaling factors are derived as the solution to a least squares minimization problem, and improvements are then made by hill climbing. The method is applied to disambiguating sentences in the Air Travel Information System corpus, and the performance of the resulting scaling factors is compared with hand-tuned factors. We then focus on one class of preference function, those based on semantic lexical collocations. Experimental results are presented showing that such functions vary considerably in selecting correct analyses. In particular, we define a function that performs significantly better than ones based on mutual information and likelihood ratios of lexical associations.