Scalable object encoding using multiplicative multilinear inter-camera prediction in the context of free view 3D video

Abstract

Recent advancements in 3D television allow for the capture of scene depth from multiple cameras and the interactive selection of view point and direction within a certain range, the so-called Free Viewpoint Video (FVV). State-of-the-art video codecs such as H.264/AVC exploit the large amount of inter-view statistical dependencies by combined temporal and inter-view prediction, i.e. prediction from temporally neighboring images as well as from images in adjacent views. This is known as Multi-view Video Coding (MVC). We propose herein an alternative object oriented video coding scheme for multi-view video with associated multiple depth data (N-video plus N-depth). A structure that we call a Multi-view Video Plane (MVP) is introduced. Object planes associated with a certain view are approximated as multilinear components of an image that are projected onto other views in a tensor-like fashion. The order of the tensor equals the number of multiple views. The coefficients of the tensor subspace projections as well as the updates of the multi-linear components (object-planes) are quantized and transmitted in the MPEG stream. Motion-compensated prediction is carried out in order to transmit the residual object planes (P-frames) using conventional MPEG algorithms. © 2012 IFIP International Federation for Information Processing.