End-to-end learning interpolation for object tracking in low frame-rate video

Abstract

In many scenarios, where videos are transmitted through bandwidth-limited channels for subsequent semantic analytics, the choice of frame rates has to balance between bandwidth constraints and analytics performance. Faced with this practical challenge, this study focuses on enhancing object tracking at low frame rates and proposes a learning Interpolation for tracking framework. This framework embeds an implicit video frame interpolation sub-network, which is concatenated and jointly trained with another object tracking sub-network. Once a low frame-rate video is an input, it is first mapped into a high frame-rate latent video, based on which the tracker is learned. Novel strategies and loss functions are derived to ensure the effective end-to-end optimisation of the authors' network. On several challenging benchmarks and settings, their method achieves a highly competitive tradeoff between frame rate and tracking accuracy. As is known, the implications of interpolation on semantic video analytics and tracking remain unexplored, and the authors expect their method to find many applications in mobile embedded vision, Internet of Things and edge computing.