Automatic subtitling for live 3D TV transmissions by real-time analysis of spatio-temporal depth map of the scene

Abstract

In order to maximize experience and perception of the 3D TV transmission, there is a rule of thumb for the 3D video content subtitling which states that the subtitle should appear in front of the in-focus content at all times of the subtitle exposure. The main problem with live 3D transmissions containing subtitles, such as TV news or football matches, is that besides a pure text and a pair of video streams acquired by a stereo rig, there must be some additional information calculated which would allow to settle the correct subtitle depth. Therefore, either all set-top-boxes must determine this depth by themselves or the broadcaster must calculate and provide this information in the Disparity Signalling Segment (DSS). In this paper we present an algorithm for automatic subtitle depth estimation based on unsupervised spatiotemporal analysis of stereoscopic pair of compressed video streams. The proposed algorithm first analyzes the texture in the streams for left and right eye in the area where the subtitle should appear. The result of this analysis is a set of correspondences, that is pairs of points corresponding to the same single point in the scene. Every correspondence yields a stereoscopic parallax vector, and the magnitude of this vector is inversely proportional to the depth of point in the scene. It is shown how to effectively calculate the depth of the subtitle from depth maps for every stereoscopic pair of frames in which this subtitle should to appear. Also, latency problems and hardware aspects of low-cost FPGA implementation of the algorithm are discussed.