Decision controller for object tracking with deep reinforcement learning

Abstract

There are many decisions which are usually made heuristically both in single object tracking (SOT) and multiple object tracking (MOT). Existing methods focus on tackling decision-making problems on special tasks in tracking without a unified framework. In this paper, we propose a decision controller (DC) which is generally applicable to both SOT and MOT tasks. The controller learns an optimal decision-making policy with a deep reinforcement learning algorithm that maximizes long term tracking performance without supervision. To prove the generalization ability of DC, we apply it to the challenging ensemble problem in SOT and tracker-detector switching problem in MOT. In the tracker ensemble experiment, our ensemble-based tracker can achieve leading performance in VOT2016 challenge and the light version can also get a state-of-the-art result at 50 FPS. In the MOT experiment, we utilize the tracker-detector switching controller to enable real-time online tracking with competitive performance and 10 × speed up.