Support Vector Machine Based Models with Sparse Auto-encoder Based Features for Classification Problem

Abstract

Auto-encoder is a special type of artificial neural network (ANN) that is used to learn informative features from data. In the literature, the generalization performance of several machine learning models have been improved either using auto-encoder based features or high dimensional features (original + auto-encoder based features). Random vector functional link (RVFL) network also uses two type of features, i.e., original features and randomized features, that makes it a special randomized neural network. These hybrid features improve the generalization performance of the RVFL network. In this paper, we introduce the idea of using additional features into robust energy-based least squares twin support vector machines (RELS-TSVM) and least squares twin support vector machines (LSTSVM). We used sparse auto-encoder with L1 norm regularization to learn the auxiliary feature representation from original feature space. These new additional features are concatenated with the original features to get the extended feature space. The conventional RELS-TSVM and LSTSVM are trained over new extended feature space. Experiments demonstrate that auto-encoder based features improve the generalization capability of the conventional RELS-TSVM and LSTSVM models. To examine the performance of the proposed classifiers, i.e., extended-RELS-TSVM (ext-RELS-TSVM) and extended LSTSVM (ext-LSTSVM), experiments have been conducted over 15 UCI binary datasets and the results show that the proposed classifiers have better generalization performance than the baseline classifiers.