From maxout to channel-out: Encoding information on sparse pathways

Abstract

Motivated by an important insight from neural science that "functionality is determined by pathway", we propose a new deep network framework, called "channel-out network", which encodes information on sparse pathways. We argue that the recent success of maxout networks can also be explained by its ability of encoding information on sparse pathways, while channel-out network does not only select pathways at training time but also at inference time. From a mathematical perspective, channel-out networks can represent a wider class of piece-wise continuous functions, thereby endowing the network with more expressive power than that of maxout networks. We test our channel-out networks on several well-known image classification benchmarks, achieving new state-of-the-art performances on CIFAR-100 and STL-10. © 2014 Springer International Publishing Switzerland.