Few-to-Many : Incremental Parallelism for Reducing Tail Latency in Interactive Services

  • Haque M
  • Eom Y
  • He Y
 et al. 
  • 57


    Mendeley users who have this article in their library.
  • 9


    Citations of this article.


Interactive services, such as Web search, recommendations, games, and finance, must respond quickly to satisfy customers. Achieving this goal requires optimizing tail (e.g., 99th+ percentile) latency. Although every server is multicore, parallelizing individual requests to reduce tail latency is challenging because (1) service demand is unknown when requests arrive; (2) blindly parallelizing all requests quickly oversubscribes hardware resources; and (3) parallelizing the numerous short requests will not improve tail latency. This paper introduces Few-to-Many (FM) incremental parallelization, which dynamically increases parallelism to reduce tail latency. FM uses request service demand profiles and hardware parallelism in an offline phase to compute a policy, represented as an interval table, which specifies when and how much software parallelism to add. At runtime, FM adds parallelism as specified by the interval table indexed by dynamic system load and request execution time progress. The longer a request executes, the more parallelism FM adds. We evaluate FM in Lucene, an open-source enterprise search engine, and in Bing, a commercial Web search engine. FM improves the 99th percentile response time up to 32% in Lucene and up to 26% in Bing, compared to prior state-of-the-art parallelization. Compared to running requests sequentially in Bing, FM improves tail latency by a factor of two. These results illustrate that incremental parallelism is a powerful tool for reducing tail latency.

Author-supplied keywords

  • dynamic parallelism
  • interactive services
  • mul-
  • tail latency
  • thread scheduling
  • tithreading
  • web search

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Md E. Haque

  • Yong hun Eom

  • Yuxiong He

  • Sameh Elnikety

  • Ricardo Bianchini

  • Kathryn S. McKinley

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free