Model-based GUI automation

Abstract

This paper addresses fundamental limitations in visual GUI automation by redefining current approaches as process-based and introducing a novel model-based alternative. By addressing the inherent stochasticity in GUI environments through insights from software testing, machine vision, robotics, and human cognition, I present a theoretical framework that enables integration and unit testing of GUI automation and significantly improves automation robustness. The approach reduces code complexity and supports larger automation applications, potentially enabling GUI automation in previously impractical domains.