Finding and removing performance bottlenecks in large systems

Abstract

Software systems obey the 80/20 rule: aggressively optimizing a vital few execution paths yields large speedups. However, finding the vital few paths can be difficult, especially for large systems like web applications. This paper describes a novel approach to finding bottlenecks in such systems, given (possibly very large) profiles of system executions. In the approach, for each kind of profile (for example, calltree profiles), a tool developer implements a simple profile interface that exposes a small set of primitives for selecting summaries of profile measurements and querying how summaries overlap. Next, an analyst uses a search tool, which is written to the profile interface and thus independent of the kind of profile, to find bottlenecks. Our search tool (Bottlenecks) manages the bookkeeping of the search for bottlenecks and provides heuristics that automatically suggest likely bottlenecks. In one case study, after using Bottlenecks for half an hour, one of the authors found 14 bottlenecks in IBM’s WebSphere Application Server. By optimizing some of these bottlenecks, we obtained a throughput improvement of 23% on the Trade3 benchmark. The optimizations include novel optimizations of J2EE and Java security, which exploit the high temporal and spatial redundancy of security checks.