Deep Correlation Multimodal Neural Style Transfer

Abstract

Style transfer is a well-known approach used to transfer the art style of a style image to an input content image, and the core method of the style transfer is to use the Gram matrix for representing the style features of images. In this paper, we investigate the advantages and disadvantages of using the Gram matrix and introduce several alternatives. In addition, we propose an end-to-end multimodal style transfer network, called deep correlation multimodal style transfer (DeCorMST), which automatically generates multiple images from a single pair of content and style images at once. We introduce deep correlation loss that integrates style losses from different correlation methods, allowing the proposed network to transfer the style of the source to the input content image in different manners. We qualitatively and quantitatively experiment with and compare DeCorMST outputs, we prove that the Gram matrix generated images is more efficient in balancing performance content preservation and style adaptation compared to the other correlations. Source code is available at https://github.com/ichirokira/CorrelationNeuralStyleTransfer.