More than Words: Using Token Context to Improve Canonicalization of Historical German

Abstract

Historical text presents numerous challenges for contemporary natural language processing techniques. In particular, the absence of consistent orthographic conventions in historical text presents difficulties for any system requiring reference to a fixed lexicon accessed by orthographic form, such as information retrieval systems, part-of-speech taggers, simple word stemmers, or more sophisticated morphological analyzers.