Inferring dataflow properties of user defined table processors

Abstract

In SCOPE, a SQL style cloud-level data-mining scripting language, table processing capabilities are often provided by user defined .NET methods. The SCOPE compiler can optimize a query plan if it knows certain dataflow relations between the input and output tables, such as column independence, column equality, or that a column's values are non-null. This paper presents an automated analysis for inferring such relations from implementations of SCOPE table processing methods. Since most table processing methods are written as .NET iterators, our analysis must accurately deal with the resulting state-machine implementing such iterators. Other complications addressed are naming and estimating column numbers, aliasing and escaping, and the inference of universally quantified loop invariants. We prototyped the analysis as Scooby, a static analyzer for .NET iterators. Scooby is able to discover useful properties for typical SCOPE programs automatically and efficiently. © 2009 Springer.