The plenoptic video

Abstract

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic video. It is a simplified light field for dynamic environments, where user viewpoints are constrained to the camera plane of a linear array of video cameras. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. The system consists of a camera array of eight Sony CCX-Z11 CCD cameras and eight Pentium 4 1.8-GHz computers connected together through a 100 Base-T local area network. It is possible to perform software-assisted real-time MPEG-2 compression at a resolution of (720 × 480). Using selective transmission, we are able to stream continuously plenoptic video with (256 × 256) resolution at a rate of 15 f/s over the network. For rendering from raw data on the hard disk, real-time rendering can be achieved with a resolution of (720 × 480) and a rate of 15 f/s. A new compression algorithm using both temporal and spatial predictions is also proposed for the efficient compression of the plenoptic videos. Experimental results demonstrate the usefulness of the proposed parallel processing based system in capturing and rendering high-quality dynamic image-based representations using off-the-shelf equipment, and its potential applications in visualization and immersive television systems. © 2005 IEEE.