A MATHEMATICAL PERSPECTIVE ON TRANSFORMERS

Abstract

Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing transformers based on their interpretation as interacting particle systems, with a particular emphasis on long-time clustering behavior. Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.