Hindsight: Evaluate video bitrate adaptation at scale

Abstract

The Adaptive bitrate algorithm (ABR) is an essential part of any HTTP-based video streaming service. Given the endless array of network environments, device capabilities, and content properties in a commercial setting, perfecting ABR remains challenging. To identify shortcomings effectively at a large scale, a scalable methodology is needed to evaluate ABR algorithms under various scenarios. The state-of-the-art method is to evaluate a production ABR retrospectively with an optimal ABR algorithm. However, optimal ABR is an NP-hard problem and therefore is costly to be deployed at a commercial scale. As a result, shortcomings in the field are often identified through manual inspection. The process is labor-intensive and often relies on experience and intuitions built from reviewing the characteristics of a large number of sessions. Motivated by our operational experience, in this paper we propose an efficient approximation for the optimal ABR problem, thus enabling large-scale deployment and benchmarking of production ABR algorithms. The contribution of the paper is two-fold. First, we provide a comprehensive study on the complexity of the optimal ABR problem, providing a compass to navigate the design space of the approximation algorithms. Second, we propose Hindsight, a linear-time and linear-space greedy algorithm that approximates the optimal solution within a reasonable error bound. This novel approach allows Hindsight to be computed and deployed at Netflixa large-scale video streaming service, providing a tool to identify sessions with suboptimal ABR performance. This task was previously infeasible at scale due to its high computational complexity. While Hindsight provides a promising methodology, many questions remain unanswered. We hope the discussion in this work can draw attention from the community and help further advance the understanding of this area.