Undecidability of the state complexity of composed regular operations

Abstract

We consider the regularity-preserving operations of intersection and marked catenation and construct an infinite sequence Ci, i = 1, 2,..., of compositions formed from the two operations. We construct also an infinite sequence of polynomials Si, i = 1, 2,..., with positive integer coefficients. As a main result we prove that it is undecidable whether or not Si is a state complexity function of Ci. All languages needed are over a fixed alphabet with at most 50 letters. We also consider some implications and generalizations, as well as present some open problems. © 2011 Springer-Verlag.