Restore Globally, Refine Locally: A Mask-Guided Scheme to Accelerate Super-Resolution Networks

Abstract

Single image super-resolution (SR) has been boosted by deep convolutional neural networks with growing model complexity and computational costs. To deploy existing SR networks onto edge devices, it is necessary to accelerate them for large image (4K) processing. The different areas in an image often require different SR intensities by networks with different complexity. Motivated by this, in this paper, we propose a Mask Guided Acceleration (MGA) scheme to reduce the computational costs of existing SR networks while maintaining their SR capability. In our MGA scheme, we first decompose a given SR network into a Base-Net and a Refine-Net. The Base-Net is to extract a coarse feature and obtain a coarse SR image. To locate the under-SR areas in the coarse SR image, we then propose a Mask Prediction (MP) module to generate an error mask from the coarse feature. According to the error mask, we select K feature patches from the coarse feature and refine them (instead of the whole feature) by Refine-Net to output the final SR image. Experiments on seven benchmarks demonstrate that our MGA scheme reduces the FLOPs of five popular SR networks by 10%–48% with comparable or even better SR performance. The code is available at https://github.com/huxiaotaostasy/MGA-scheme.