Cross Modal Facial Image Synthesis Using a Collaborative Bidirectional Style Transfer Network

Abstract

In this paper, we present a novel collaborative bidirectional style transfer network based on generative adversarial network (GAN) for cross modal facial image synthesis, possibly with large modality gap. We think that representation decomposed into content and style can be effectively exploited for cross modal facial image synthesis. However, we have observed that unidirectional application of decomposed representation based style transfer in case of large modality gap does not work well for this purpose. Unlike existing image synthesis methods that typically formulate image synthesis as an unidirectional feed forward mapping, our network utilizes mutual interaction between two opposite mappings in a collaborative way to address complex image synthesis problem with large modality gap. The proposed bidirectional network aligns shape content from two modalities and exchanges their appearance styles using feature maps of the layers in the encoder space. This allows us to effectively retain the shape content and transfer style details for synthesizing each modality. Focusing on facial images, we consider facial photo, sketch, and color-coded semantic segmentation as different modalities. The bidirectional synthesis results for the pairs of these modalities show the effectiveness of the proposed approach. We further apply our network to style-content manipulation to generate multiple photo images with various appearance styles for a same content shape. The proposed method can be adopted for solving other cross modal image synthesis tasks. The dataset and source code are available at https://github.com/kamranjaved/Bidirectional-style-transfer-network.