A flexible and efficient ML lexer tool based on extended regular expression submatching

Abstract

Lexical analysis has many applications beyond the first phase of compilation in programming language processing. We argue that extended regular expressions combined with the ability to extract submatch information significantly increase the expressiveness of lexer specifications. We show that such an expressive lexical analysis can be done efficiently using some novel automata-based methods. The approach has been implemented in an ML lexer tool which is compatible with ocamllex. Experimental results confirm that our approach is competitive with respect to existing ML lexer tools. © 2014 Springer-Verlag.