Video based live tracking of fishes in tanks

Abstract

We explore video tracking and classification in the context of real time marine wildlife observation. Among other applications it can help biologists by automating the process of gathering data, which is often done manually. In this paper we present a system to tackle the challenge of tracking and classifying fish in real time. We apply Background Subtraction techniques to detect the fish, followed by Feature Matching methods to track their movements over time. To deal with the shortcomings of tracking by detection we use a Kalman Filter to predict fish positions and a local search recovery method to re-identify fish tracks that are temporarily lost due to occlusions or lack of contrast. The species of tracked fish is recognized through Image Classification methods, using environment dependent features. We developed and tested our system using a custom built dataset, with several labeled image sequences of the fish tanks in the Oceanário de Lisboa. The impact of the proposed tracking methods are quantified and discussed. The proposed system is able to track and classify fish in real time in two scenarios, main tank and coral reef, reflecting different challenges.