Weighted pooling based on visual saliency for image classification

Abstract

A selection of pooling region is an important issue on image classification process. Although Spatial Pyramid Matching (SPM) simply selects pooling regions by partitioning an image into rectangular grids, the SPM has been used the most until now to exploit the spatial information of features. In this paper, in addition to the SPM, we concatenate a novel pooling method which decomposes an image into an object region and a background region. Since the regions deal with not only the spatial information but also objectness that is higher level knowledge than the simple rectangular grids, flexible sub-regions can be achieved with our method. In order to estimate object region from an unlabeled image, we adopt the visual saliency map for a probabilistic measure of objectness. To do this, our soft region pooling replaces the existing max pooling, which helps features inside the probabilistic boundary of object/background region to be selected. Beyond applying the objectness score only to select features for the max pooling, weight for each element of image representation vector is also applied to emphasize an important component vector. The proposed method is implemented based on LLC, and tested on widely used dataset, Caltech-101 and Caltech-256. The proposed method achieves 2.77% improvement on Caltech-101 and achieves 1.84% improvement on Caltech- 256 compared with the accuracy of LLC.