Skip to content
Journal article

Controlled experiments on the web: survey and practical guide

Kohavi R, Longbotham R, Sommerfield D, Henne R, Kohavi R, Longbotham R, Sommerfield D, Henne R ...see all

Data Min Knowl Disc, vol. 18 (2009) pp. 140-181

  • 5


    Mendeley users who have this article in their library.
  • N/A


    Citations of this article.
  • N/A


    ScienceDirect users who have downloaded this article.
Sign in to save reference


The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for estab-lishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that sig-nificant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We pro-vide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limita-tions (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for vari-ance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled Responsible editor: R. Bayardo. Controlled experiments on the web 141 experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the out-come of interest, leading to new hypotheses and creating a virtuous cycle of improve-ments. Organizations that embrace controlled experiments with clear evaluation cri-teria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

Author-supplied keywords

  • Multi
  • Testing · MVT
  • Variable
  • Website optimization ·

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in


  • Ron Kohavi

  • Roger Longbotham

  • Dan Sommerfield

  • Randal M Henne

  • R Kohavi

  • R Longbotham

Cite this document

Choose a citation style from the tabs below