Impact of etcd deployment on Kubernetes, Istio, and application performance

Abstract

This experience article describes lessons learned as we conducted experiments in a Kubernetes-based environment, the most notable of which was that the performance of both the Kubernetes control plane and the deployed application depends strongly and in unexpected ways on the performance of the etcd database. The article contains (a) detailed descriptions of how networking with and without Istio works in Kubernetes, based on the Flannel Container Networking Interface (CNI) provider in VXLAN mode with IP Virtual Server (IPVS)-backed Kubernetes Services, (b) a comprehensive discussion about how to conduct load and performance testing using a closed-loop workload generator, and (c) an open source experiment framework useful for executing experiments in a shared cloud environment and exploring the resulting data. It also shows that statistical analysis may reveal the data resulting from such experiments to be misleading even when careful preparations are made, and that nondeterministic behavior stemming from etcd can affect both the platform as a whole and the deployed application. Finally, it is demonstrated that using high-performance backing storage for etcd can reduce the occurrence of such nondeterministic behaviors by a statistically significant (P