Interpreting MOS scores, when can users see a difference? Understanding user experience differences for photo quality

Abstract

The use of no-reference image quality evaluation tools that produce MOS scores, like the VIQET tool which was released by the Video Quality Expert Group, raises the question of whether the produced MOS differences between images correspond to noticeable differences in quality by the consumers. In this work, we attempted to approximate the minimum MOS difference that is required in order for people to be able to distinguish between a higher and a lower quality image under realistic conditions that are commonly encountered in the current consumer space. 91 people participated in a subjective just-noticeable-differences study across three countries that used non-simulated image stimuli, produced and evaluated through crowd sourcing for the validation of the VIQET no-reference image quality tool. The image dataset consisted of 15 different scenes belonging to three different scene types, with a total of 210 different image pairs being used. After evaluating the quality of the collected data, a logistic regression analysis approach was employed in order to estimate the minimum MOS difference required between two images in order for a given percentage of people to be able to detect the higher quality image.