Enhanced sketchnoting through semantic integration of learning material

Abstract

Within the transition from pen-and-paper to digital solutions, many tablet-based note-taking apps have been developed. In addition, sketchnoting, a highly visual form of note-taking, has gained in popularity, especially in teaching and learning. While current solutions already reduce the burden of organizing and searching through stacks of paper, they provide little support in finding additional information related to the notes, and search capabilities are mostly limited to textual queries and content. In this paper, we present a novel solution for digital sketchnoting aimed at enhancing the learning experience of students. In addition to common capabilities such as handwriting recognition, our note-taking app integrates semantic annotation and drawing recognition. Handwritten notes are recognized and passed through concept recognition and entity linking tools to enable a knowledge graph-based integration of contextually relevant learning resources. We extend traditional search capabilities by including the semantic metadata from the related material as well as enabling visual queries to find recognized sketches. Finally, resembling the exchange of paper notes among students, our app allows to share the semantically enhanced notes with other devices.