Fast rate-distortion optimization in H.264/AVC video coding

Abstract

One of the new features in the H.264/AVC encoder is the use of Lagrangian Rate-Distortion Optimised (RDO) mode decision at the macroblock (MB) level. This brute-force algorithm searches through the 10 different MB coding modes to obtain the best one for encoding that MB, and is hence computationally expensive. This paper proposes a novel algorithm where the RDO can be reorganized in a better order such that the most likely MB modes will be tried first, and an early termination of the RDO process will be used once the calculated rate-distortion cost (RDcost) is below a preset threshold. The preset threshold is dependent on the RDcost of previous and neighbouring MBs that have been coded. This is based on the observation that the RDcost of an MB in the current frame is highly correlated to a co-located MB in the previous frame. Experimental results have shown that the new algorithm has dramatically reduced the encoding times of up to 61.8%, with negligible increases in bitrates or PSNR. © Springer-Verlag Berlin Heidelberg 2005.