The Impact of System Transparency on Analytical Reasoning

Abstract

In this paper, we present the hypothesis that system transparency is critical for tasks that involve expert sensemaking. Artificial Intelligence (AI) systems can aid criminal intelligence analysts, however, they are typically opaque, obscuring the underlying processes that inform outputs, and this has implications for sensemaking. We report on an initial study with 10 intelligence analysts who performed a realistic investigation exercise using the Pan natural language system [10, 11], in which only half were provided with system transparency. Differences between conditions are analysed and the results demonstrate that transparency improved the ability of analysts to reason about the data and form hypotheses.