Automatic generation of dictionaries: The journalistic lexicon case

Abstract

Text normalisation is an important task in the context of Natural Language Processing. By normalisation, free text is mapped into dictionaries, i.e. indexed collections of locutions recognised as typical of a particular jaergon. In general, technical dictionaries are difficult to build and validate. They are typically constructed by hand on the basis of everyday human work and they are agreement-based. This is indubitably time consuming and the approach requires a strong human supervision and does not provide a general methodology. In this paper, we perform the first steps towards the to automatic building of a dictionary for Italian journalistic lexicon, called NewsDict, based on sub dictionaries able to characterise main topics occurring in newspaper articles. We exploit a dataset of annotated documents from some Italian newspapers and a statistical techniques based on the Mutual Information Principle. Documents contains information such as the release date and the topic of the article and has been directly annotated by the author. To check the accuracy of the dictionary we built, we develop an initial test. We normalise a control set of journal article into NewsDict. Crossing results presented in this paper against the human annotation, we provide a fist measure of performances of the described methodology.