TRANSACTIONS ON IMAGE PROCESSING 1 Optimizing a Tone Curve for Backward-Compatible High Dynamic Range Image and Video Compression Zicong Mai, Student Member, IEEE, Hassan Mansour, Student Member, IEEE, Rafal Mantiuk, Panos Nasiopoulos, Member, IEEE, Rabab Ward, Fellow, IEEE, and Wolfgang Heidrich Abstract���For backward compatible high dynamic range (HDR) video compression, the HDR sequence is reconstructed by inverse tone-mapping a compressed low dynamic range (LDR) version of the original HDR content. In this paper, we show that the appropriate choice of a tone-mapping operator (TMO) can significantly improve the reconstructed HDR quality. We develop a statistical model that approximates the distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected mean square error (MSE) in the reconstructed HDR sequence. We also develop a simplified model that reduces the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs. Index Terms���high dynamic range imaging, bit-depth scalable, tone-mapping, HDR video compression. I. INTRODUCTION Natural scenes contain far more visible information than can be captured by the majority of digital imagery and video devices. This because traditional display devices can only sup- port a limited dynamic range (contrast) and color gamut. New display and projection technologies, however, employ narrow- wavelength LED light sources that expand the boundaries of the displayable color gamut. This expansion will be again vastly enlarged with the next generation display technologies that will employ dual modulation [1] or backlight dimming that enhance intra- and inter-frame contrast. For video compression, these advances in display technolo- gies have motivated the use of extended gamut color spaces. These include xvYCC (x.v.Color) for home theater [2] and the Digital Cinema Initiative color space for digital theater applications. Yet, even these extended color spaces are too limited for the amount of contrast that can be perceived by the human eye. High dynamic range (HDR) video encoding goes beyond the typical color space restrictions and attempts to encode all colors that are visible and distinguishable to the human eye [3], and is not restricted by the color gamut of the display technology used. The main motivation is to create Zicong Mai, Hassan Mansour, Panos Nasiopoulos, and Rabab Ward are with the Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC, Canada (email: zicongm@ece.ubc.ca, hassanm@ece.ubc.ca, panosn@ece.ubc.ca, and rababw@ece.ubc.ca). Rafal Mantiuk is with the School of Computer Science, Bangor University, North Wales, United Kingdom (email: mantiuk@bangor.ac.uk). Wolfgang Heidrich is with the Department of Computer Science, University of British Columbia, Vancouver, BC, Canada (email: heidrich@cs.ubc.ca). Fig. 1. General structure of the scalable approach used for backward- compatible HDR video encoding. The base layer encodes an 8-bit LDR rep- resentation of the HDR input. The enhancement layer encodes the difference (residual) between the inverse tone-mapped base layer and the original HDR source. a video format that would be future-proof, independent of a display technology, and limited only by the performance of the human visual system (HVS). HDR images preserve colorimetric or photometric pixel values (such as CIE XYZ) within the visible color gamut and allows for intra-frame contrast exceeding 5-6 orders of magnitude (106 : 1), without introducing contouring, banding or posterization artifacts caused by excessive quantization. The photometric or colorimetric values, such as luminance (cd �� m-2) or spectral radiance (W��sr-1��m-3), span much larger range of values than luma and chroma values (gamma corrected) used in typical video encoding (JPEG, MPEG, etc.). The obvious representation for the colorimetric values are floating point numbers, which, however, are impractical for image and video coding applications. For that reason several HDR color encodings and file formats have been proposed, including the Radiance RGBE (.hdr) [4], OpenEXR (.exr) [5] and LogLuv TIFF (.tiff) [6] file formats. They employ either more efficient floating point coding (OpenEXR and RGBE) or perceptually motivated compressive functions (LogLuv and [3]), which extends a typical ���gamma correction��� to the entire range of luminance values. Although HDR image and video encoding offers truly device-independent representation, the majority of existing digital display devices can only support 8-bit video content. Therefore, high dynamic range video formats are unlikely to be broadly accepted without the backward-compatibility with these devices. Such backward-compatibility can be achieved if the HDR video stream contains 1) a backward-compatible