GUI Element Detection from Mobile UI Images Using YOLOv5

Abstract

In mobile application development, building a consistent user interface (UI) might be a costly and time-consuming process. This is especially the case if an organization has a separate team for each mobile platform such as iOS and Android. In this regard, the companies that choose the native mobile app development path end up going through do-overs as the UI work done on one platform needs to be repeated for other platforms too. One of the tedious parts of UI design tasks is creating a graphical user interface (GUI). There are numerous tools and prototypes in the literature that aim to create feasible GUI automation solutions to speed up this process and reduce the labor workload. However, as the technologies evolve and improve new versions of existing algorithms are created and offered. Accordingly, this study aims to employ the latest version of YOLO, which is YOLOv5, to create a custom object detection model that recognizes GUI elements in a given UI image. In order to benchmark the newly trained YOLOv5 GUI element detection model, existing work from the literature and their data set is considered and used for comparison purposes. Therefore, this study makes use of 450 UI samples of the VINS dataset for testing, a similar amount for validation and the rest for model training. Then the findings of this work are compared with another study that has used the SSD algorithm and VINS dataset to train, validate and test its model, which showed that proposed algorithm outperformed SSD’s mean average precision (mAP) by 15.69%.