Trust in imperfect automation

Abstract

The types of unreliability that an automated system may express can have effects on a user’s perception of that automation’s overall operational ability. A software program which makes one type of mistake might be judged more harshly than another program which makes a different sort of error; even if both have equal success rates. Here I use a Hidden Object Game to examine people’s different responses to a program when it appears to either miss its target objects or, alternatively, to make false alarms. Playing at both high and low clutter levels, participants who believed they were working with an automated system which missed targets decreased their trust in that automation, and judged its performance more harshly, compared to participants who believed the automation was making false alarms. Participants in the combined low clutter and miss condition showed the strongest decrease in trust. When asked to guess how often the program had been correct this group also gave it the lowest mean score. These results demonstrates that in a target detection task, automation that misses targets will be judged more harshly than automation that errs on the side of false alarms.