Activity Segmentation and Fish Tracking From Sonar Videos by Combining Artifacts Filtering and a Kalman Approach

Abstract

Ecosystems are highly dynamic systems that are constantly changing under the influence of a variety of external factors. This is especially true for marine ecosystems, which are under multiple stresses. The cumulative effects of overexploitation, on the one hand, and the simultaneous manifestation of anthropogenic climate change, on the other, mean that fish stocks are the most endangered components of marine ecosystems. To minimize these vulnerabilities to marine ecosystems and ensure natural and sustainable resource use, monitoring systems must be placed in oceans and seas. Examples of the development of these monitoring systems are provided by the Underwater Fish Observatory (UFO) and UFOTriNet, two projects being conducted by several researchers from marine biology, engineering, and industry in Germany between 2014 and 2016 and between 2019 and 2023, respectively. The systems collect abiotic as well as camera and sonar data to count and analyze fish populations over the seasons. This work proposes a method for robust fish counting using sonar data, supplemented by camera data. To successfully accomplish this task, activity segmentation and object tracking are important steps. Background subtraction is often used as a pre-processing step for stationary sonars. Our proposed method improves this step by bandpass filtering considering the motion of all actors in the sonar data. For the segmentation step, our method uses a simple Gaussian distribution model with positional covariances which are computed directly from the intensity image. The tracking step is performed using a classical Kalman filter which estimates the velocity and position of each object in Cartesian coordinates. Sonar detections in close range of the observation area are compared with camera detections for validation. In addition, automated parameter optimization is used to maximize the correlation with the camera detections. Furthermore, the proposed method is applied to the Caltech fish counting dataset and compared with a deep learning method based on YOLOv5. While YOLO is still superior in detection and counting metrics, the multi object tracking accuracy is somewhat higher with our method.