The evolution of user experiences and interfaces for delivering context-aware recommendations to information analysts

Abstract

Rigor in the products of information analysis is essential for decision makers to rely on the assessments contained within them. Zelik, Patterson, and Woods [1] defined an eight-attribute metric for communicating the rigor of analytic products. This paper describes two iterations of the process of designing, implementing, and evaluating a context-aware web application that uses this analytic rigor metric to recommend augmentations to analysts’ workflow that will improve the quality of the resultant products. We used multiple methods to evaluate this tool with subject matter experts, including brainstorming, collaborative card sorting, semi-structured interviews, cognitive walkthroughs, and heuristic evaluations. This research found that: (1) it is critical to have flexible recommendations that adapt to movements between foraging and sense-making components of workflow, and the changing structure of the analysis; and (2) persistent visualizations of analytic rigor assessments are distracting, and promote interpretation as a performance metric rather than a process aid.