Generalized bayesian canonical correlation analysis with missing modalities

Abstract

Multi-modal learning aims to build models that can relate information from multiple modalities. One challenge of multi-modal learning is the prediction of a target modality based on a set of multiple modalities. However, there are two challenges associated with the goal: Firstly, collecting a large, complete dataset containing all required modalities is difficult; some of the modalities can be missing. Secondly, the features of modalities are likely to be high dimensional and noisy. To deal with these challenges, we propose a method called Generalized Bayesian Canonical Correlation Analysis with Missing Modalities. This method can utilize the incomplete sets of modalities. By including them in the likelihood function during training, it can estimate the relationships among the non-missing modalities and the feature space in the non-missing modality accurately. In addition, this method can work well on high dimensional and noisy features of modalities. This is because, by a probabilistic model based on the prior knowledge, it is strong against outliers and can reduce the amount of data necessary for the model learning even if features of modalities are high dimensional. Experiments with artificial and real data demonstrate our method outperforms conventional methods.