Detecting content-bearing words by serial clustering - extended abstract

Abstract

Information Retrieval Systems typically distinguish between content bearing words and terms on a stop list. But 'content-bearing' is relative to a collection. For optimal retrieval efficiency, it is desirable to have automated methods for custom building a stop list. This paper defines the notion of serial clustering of words in text, and explores the value of such clustering as an indicator of a word bearing content. The numerical measures we propose may also be of value in assigning weights to terms in requests. Experimental support is obtained from natural text databases in three different languages.