The generating function of ternary trees and continued fractions

Abstract

Michael Somos conjectured a relation between Hankel determinants whose entries 1/2n+1(3nn) count ternary trees and the number of certain plane partitions and alternating sign matrices. Tamm evaluated these determinants by showing that the generating function for these entries has a continued fraction that is a special case of Gauss's continued fraction for a quotient of hypergeometric series. We give a, systematic application of the continued fraction method to a, number of similar Hankel determinants. We also describe a simple method for transforming determinants using the generating function for their entries. In this way we transform Somos's Hankel determinants to known determinants, and we obtain, up to a power of 3, a Hankel determinant for the number of alternating sign matrices. We obtain a combinatorial proof, in terms of nonintersecting paths, of determinant identities involving the number of ternary trees and more general determinant identities involving the number of r-ary trees.