Predicting Video Frames Using Feature Based Locally Guided Objectives

Abstract

This paper presents feature reconstruction based approach using Generative Adversarial Networks (GAN) to solve the problem of predicting future frames from natural video scenes. Recent GAN based methods often generate blurry outcomes and fail miserably in case of long-range prediction. Our proposed method incorporates an intermediate feature generating GAN to minimize the disparity between the ground truth and predicted outputs. For this, we propose two novel objective functions: (a) Locally Guided Gram Loss (LGGL) and (b) Multi-Scale Correlation Loss (MSCL) to further enhance the quality of the predicted frames. LGGL aides the feature generating GAN to maximize the similarity between the intermediate features of the ground-truth and the network output by constructing Gram matrices from locally extracted patches over several levels of the generator. MSCL incorporates a correlation based objective to effectively model the temporal relationships between the predicted and ground-truth frames at the frame generating stage. Our proposed model is end-to-end trainable and exhibits superior performance compared to the state-of-the-art on four real-world benchmark video datasets.