Integrating a dynamic lexicon with a dynamic full-text retrieval system

Abstract

There has been a great deal of interest within the Information Retrieval community in evaluating the use of linguistic knowledge to improve the indexing and searching of textual databases. Such systems must often employ a lexicon to store information about the words and phrases comprising the application's domain. Unlike a static lexicon, a dynamic lexicon raises practical concerns about the coordination between the state of the lexicon and IR indexing schemes based on lexical knowledge. Additionally, it introduces a host of database management issues, many of which are similar to those found in the text databases as well. In this paper, we explore a range of system design and performance issues that arise when integrating a dynamic lexicon with a dynamic full-text information retrieval system. We observe that the principle of functional isolation argues against the use of language-dependent information in article indexes and favors the use of query-time strategies for applying lexical knowledge. We propose and evaluate a system architecture which embodies this principle. We also show how a storage and retrieval infrastructure based on Burkowski's [BURKOWSKI92] 'containment model' abstraction can be employed to implement both the text retrieval and lexicon facilities required in an integrated system.