A Meta-Learning Approach to Generating Functional Descriptions of Graphical User Interfaces

Abstract

Graphical user interfaces (GUIs) are important software engineering artifacts that commonly connect end users with software apps to fulfill requirements. While previous work extracts metadata to build a GUI's search profile and exploits deep learning to caption a GUI, little is known about how to capture the GUI's essential functionality. We shorten the gap in this paper by presenting a novel approach that leverages fewshot prompting and meta-learning. In particular, we devise four attributes of a GUI in order to instrument the meta-learner. The meta-learner is then integrated with the few-shot learner through an iteration mechanism. We comprehensively evaluate our approach with a GUI description dataset and a GUI tracing dataset. The results show that Qwen is a viable implementation option for our approach, and that our approach outperforms the baseline prompting methods.