Video Shot Boundary Detection Algorithm

Abstract

The detection of shot boundaries provides a base for nearly all video abstraction and high-level video segmentation approaches. Therefore solving the problem of shot boundary detection is one of the major prerequisites for revealing higher level video content structure. Moreover, the other research areas which can be benefitted considerably from successful automation of shot boundary detection processes are distance learning, telemedicine, interactive television, digital libraries, multimedia news, video restoration and geographical information system. Despite all the research activity in shot boundary detection, there are some issues which have not been adequately addressed and need to be resolved. The major challenges for shot boundary detection are, detection of gradual transition and elimination of the disturbances caused by abrupt illumination change and motion. The disturbances caused by abrupt illumination change is mainly due to flashlight, fire, flicker and explosion, whereas the disturbance caused by motion is due to rapid camera and object motion in the consecutive frames. It is difficult to develop a single approach which is not only invariant to various disturbances mentioned above but should also be sensitive enough to capture the details of visual content and have excellent detection performance for all types of shot boundaries. In this thesis, we focus on detection of wipes and avoiding the disturbances due to flashlight, fire, flicker, explosion and motion.Wipe transition detection in video segmentation is more difficult to detect than abrupt and other gradual transitions, due to diversity in patterns, and difficulty in distinguishing wipe from camera and object motion. An algorithm has been proposed for wipe transition detection. In the proposed algorithm, first the moving lines due to wipe are obtained, which helps in eliminating most of the edges due to object boundaries and retains true wipe boundaries. Then Hough transform is applied on these moving lines to detect and categorize various wipe types. In order to decrease the computational load of the proposed algorithm, preprocessing step has been proposed as a first stage of the algorithm. The preprocessing step consists of calculation of statistical image difference between the consecutive frames to obtain the potential wipe frames. These potential wipe frames are processed through the proposed algorithm to detect and identify wipes. The proposed algorithm detects and identifies various types of wipes and also distinguishes wipes from object and camera motion. Performance comparison of the proposed algorithm, with and without preprocessing, with the other existing techniques clearly exhibited its effectiveness in terms of better Recall, Precision, F1 measure and Detection Rate.Elimination of disturbances caused by flashlights is one of the major challenge in shot boundary detection. The existence of flashlight changes the luminance and chrominance abruptly due to sudden visual effect across the video sequence. This contributes to a larger difference between consecutive frames and leads to false detection of shot boundary. Existing flashlight detection methods have limitation in terms of duration of flashlights, nature of flashlights, computation time and calculation of an appropriate global threshold. Global threshold is a fixed threshold and needs to be adjusted for each new video. To address these issues an algorithm has been proposed for shot boundary detection due to abrupt transition in the presence of flashlight. In the proposed algorithm, the illumination effect due to flashlight was suppressed using logarithmic transform followed by discrete cosine transform, and then discrete wavelet transform based metric was used to find potential shot boundaries, and finally, local or adaptive threshold was used to declare shot boundary. Local and an adaptive threshold depends on a mean and standard deviation of the feature difference metric within a temporal window and hence overcomes the limitation of a global threshold. The proposed algorithm is tested on movie videos, and experimental results validate the effectiveness of the method to avoid false positives due to flash lights in shot boundary detection.Detection of fire in video for fire alarm systems has been studied by many researchers, but detection of shot boundaries under fire, flicker and explosion (FFE) is one of the under studied areas. In thriller movies, FFE occur more often than other special effects and lead to false detection of shot boundary. Major metrics have been tested for detection of shot boundaries under FFE for various movies. It has been observed that precision is low for almost all metrics due to false positives caused by FFE.We proposed an algorithm based on cross correlation coefficient, stationary wavelet transform, and combination of local and adaptive thresholds for detection of shot boundaries under FFE. The proposed algorithm is tested on three movies and experimental results validate the effectiveness of our method in terms of better Recall and Precision.We evaluated the performance of traditional shot boundary detection metrics in the presence of camera and object motion in RGB, HSV and Y UV color spaces for eight movies. It has been observed that all these metrics provided poor results due to disturbances caused by these motion. The maximum false positives and missed detections are due to frame difference between consecutive frames caused by fast camera motion. Developing a single approach which will eliminate the disturbances due illumination as well as motion is a challenging task. To address these challenge, we propose an algorithm for shot boundary detection in the presence of illumination and motion. In this proposed algorithm, extraction of structure feature of each frames were done using dual tree complex wavelet transform. These structure features were in sensitive to illumination and motion between consecutive frames. Then spatial domain structure similarity algorithm was applied on these structure features to find potential shot boundaries, finally correct shot boundaries are declared by using local and adaptive thresholds. The performance comparison of the proposed algorithm with other existing techniques, validate its effectiveness in terms of better Recall, Precision and F1 score.