Auto-extraction and integration of metrics for web user interfaces

Abstract

Metric-based assessment of web user interface (WUI) quality attributes is shifting from code (HTML/CSS) analysis to mining webpages'visual representations based on image recognition techniques. In our paper, we describe a visual analysis tool which takes a WUI screenshot and produces structured and machine-readable representation (JSON) of the interface elements' spatial allocation. The implementation is based on OpenCV (image recognition functions), dlib (trained detector for the elements' classification), and Tesseract (label and content text recognition). The JSON representation is used to automatically calculate several metrics related to visual complexity, which is known to have major effect on user experience with UIs. We further describe a WUI measurement platform that allows integration of the currently dispersed sets of metrics from different providers and demonstrate the platform's use with several remote services. We perform statistical analysis of the collected metrics in relation to complexity-related subjective evaluations obtained from 63 human subjects of various nationalities. Finally, we build predictive models for visual complexity and show that their accuracy can be improved by integrating the metrics from different sets. Regressions with the single index of visual complexity metric that we proposed had R 2 =0.460, while the best joint model with 4 metrics had R 2 =0.647.