Doodle2App: Native app code by freehand UI sketching

Abstract

User interface development typically starts with freehand sketching, with pen on paper, which creates a big gap in the software development process. Recent advances in deep neural networks that have been trained on large sketch stroke sequence collections have enabled online sketch detection that supports many sketch element classes at high classification accuracy. This paper leverages the recent Google Quick, Draw! dataset of 50M sketch stroke sequences to pre-train a recurrent neural network and retrains it with sketch stroke sequences we collected via Amazon Mechanical Turk. The resulting Doodle2App website offers a paper substitute, i.e., a drawing interface with interactive UI preview and can convert sketches to a compilable single-page Android application. On 712 sketch samples Doodle2App achieved higher accuracy than the state-of-the-art tool Teleport. A video demo is at https://youtu.be/P4sb0pKTNEY