The evaluator effect in usability tests

  • Jacobsen N
  • Hertzum M
  • John B
  • 132


    Mendeley users who have this article in their library.
  • N/A


    Citations of this article.


Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been letI unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained.

Author-supplied keywords

  • Usability
  • evaluator effect
  • usability test
  • user testing

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Niels Ebbe Jacobsen

  • Morten Hertzum

  • Bonnie E. John

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free