Structure fusion and propagation for zero-shot learning

Abstract

The key of zero-shot learning (ZSL) is how to find the information transfer model for bridging the gap between images and semantic information (texts or attributes). Existing ZSL methods usually construct the compatibility function between images and class labels with consideration of the relevance on the semantic classes (the manifold structure of semantic classes). However, the relationship of image classes (the manifold structure of image classes) is also very important for the compatibility model construction. It is difficult to capture the relationship among image classes due to unseen classes, so that the manifold structure of image classes often is ignored in ZSL. To complement each other between the manifold structure of image classes and that of semantic classes information, we propose structure fusion and propagation (SFP) for improving the performance of ZSL for classification. SFP can jointly consider the manifold structure of image classes and that of semantic classes for approximating to the intrinsic structure of object classes. Moreover, the SFP can describe the constraint condition between the compatibility function and these manifold structures for balancing the influence of the structure fusion and propagation iteration. The SFP solution provides not only unseen class labels but also the relationship of two manifold structures that encodes the positive transfer in structure fusion and propagation. Experiments demonstrate that SFP can attain the promising results on the AwA, CUB, Dogs and SUN datasets.