Synchronized audio and visual decoding scheme that is tolerant to variation of processing environment

Abstract

We propose a synchronized audio and visual (AV) decoding scheme that is tolerant of variation in the processing environment, assuming that the future AV terminal will be implemented by software. The processing environment means the set of conditions, such as workload and equipment, that defines AV processing ability. Due to its software implementation, the AV terminal might encounter fatal degradation in decoding quality when the processing environment deteriorates. To prevent this, our scheme introduces a prioritized frame packet structure (access unit) to the AV bitstream, in which the access units are selectively discarded by comparing their processing priorities with the discarding threshold. The processing priority for each access unit is determined by the relative importance of each AV stream and the relative importance of each frame within the stream. The discarding threshold is adjusted according to a decoding time stamp attached to the access unit. The proposed scheme allows us to provide the workload adjusting mechanism independently of underlying terminal implementation ; the decoding quality is gracefully degraded in the case of high workload. Experimental results show the effectiveness of our scheme against variations in the processing environment.