Multiple template detection based on segments

Abstract

Web pages contain a combination of informative contents and redundant contents which are primarily used for navigation, advertisements, copyright and decoration. Detecting templates correctly and precisely thus becomes a vital part for many applications. Methods for template detection have been studied extensively. However, they are insufficient to detect multiple templates in a Web site. In this paper, we propose a novel segment-based template detection method to identify templates. Our method works in three steps. First, for each Web site we construct a SSOM (Site-oriented Segment Object Model) tree from sampled pages in a Web collection, through aligning the pages' SOM (Segment Object Model) trees. Second, we construct a template from the SSOM tree. At last, the template can be used to detect templates for the Web site: Given a page in the Web site, its template contents are gained with mapping between its SOM tree and the SSOM tree and classifying. The proposed method is evaluated with two mining tasks, Web page clustering and classification. It leads to a significant improvement when compared to previous template detection methods. © 2014 Springer International Publishing Switzerland.