Myths and facts about the efficient implementation of finite automata and lexical analysis

Abstract

Finite automata and their application in lexical analysis play an important role in many parts of computer science and particularly in compiler constructions. We measured 12 scanners using different implementation strategies and found that the execution time differed by a factor of 74. Our analysis of the algorithms as well as run-time statistics on cache misses and instruction frequency reveals substantive differences in code locality and certain kinds of overhead typical for specific implementation strategies. Some of the traditional statements on writing "fast" scanners could not be confirmed. Finally, we suggest an improved scanner generator.