BatchNorm Decomposition for Deep Neural Network Interpretation

Abstract

Layer-wise relevance propagation (LRP) has shown potential for explaining neural network classifier decisions. In this paper, we investigate how LRP is to be applied to deep neural network which makes use of batch normalization (BatchNorm), and show that despite the functional simplicity of BatchNorm, several intuitive choices of published LRP rules perform poorly for a number of frequently used state of the art networks. Also, we show that by using the ε-rule for BatchNorm layers we are able to detect training artifacts for MobileNet and layer design artifacts for ResNet. The causes for such failures are analyzed deeply and thoroughly. We observe that some assumptions on the LRP decomposition rules are broken given specific networks, and propose a novel LRP rule tailored for BatchNorm layers. Our quantitatively evaluated results show advantage of our novel LRP rule for BatchNorm layers and its wide applicability to common deep neural network architectures. As an aside, we demonstrate that one observation made by LRP analysis serves to modify a ResNet for faster initial training convergence.