Marginal Deep Architecture: Stacking Feature Learning Modules to Build Deep Learning Models

Abstract

Recently, many deep models have been proposed in different fields, such as image classification, object detection, and speech recognition. However, most of these architectures require a large amount of training data and employ random initialization. In this paper, we propose to stack feature learning modules for the design of deep architectures. Specifically, marginal Fisher analysis (MFA) is stacked layer-by-layer for the initialization and we call the constructed deep architecture marginal deep architecture (MDA). When implementing the MDA, the weight matrices of MFA are updated layer-by-layer, which is a supervised pre-training method and does not need a large scale of data. In addition, several deep learning techniques are applied to this architecture, such as backpropagation, dropout, and denoising, to fine-tune the model. We have compared MDA with some feature learning and deep learning models on several practical applications, such as handwritten digits recognition, speech recognition, historical document understanding, and action recognition. The extensive experiments show that the performance of MDA is better than not only shallow feature learning models but also related deep learning models in these tasks.