Iterative Few-shot Semantic Segmentation from Image Label Text

Abstract

Few-shot semantic segmentation aims to learn to segment unseen class objects with the guidance of only a few support images. Most previous methods rely on the pixel-level label of support images. In this paper, we focus on a more challenging setting, in which only the image-level labels are available. We propose a general framework to firstly generate coarse masks from image label text with the help of the powerful vision-language model CLIP, and then refine the mask predictions of support and query images iteratively and mutually. During the refinement, we design an Iterative Mutual Refinement (IMR) module to adapt to the varying quality of the coarse support mask. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that our method not only outperforms the state-of-the-art weakly supervised approaches by a significant margin, but also achieves comparable or better results to recent supervised methods. Moreover, our method owns an excellent generalization ability for the images in the wild and uncommon classes. Code will be available at https://github.com/Whileherham/IMR-HSNet.