A tale of the openSSL state machine: A large-scale black-box analysis

Abstract

State machine inference is a powerful black-box analysis technique that can be used to learn a state machine implemented in a system, i.e. by only exchanging valid messages with the implementation a state machine can be extracted. In this paper we perform a large scale analysis of the state machines as implemented over the last 14 years in OpenSSL, one of the most widely used implementations of TLS, and in LibreSSL, a fork of OpenSSL. By automating the learning process, the state machines were learned for 145 different versions of both the serverside and the client-side. For the server-side this resulted in 15 unique state machines for OpenSSL and 2 for LibreSSL. For the client-side, 9 unique state machines were learned for OpenSSL and one for LibreSSL. Analysing these state machines provides an interesting insight in the evolution of the state machine of OpenSSL. Security vulnerabilities and other bugs related to their implementation can be observed, together with the point at which these are fixed. We argue that these problems could have been detected and fixed earlier if the developers would have had the tools available to analyse the implemented state machines.