Agreement behavior of isolated annotators for maintenance work-order data mining

Abstract

Maintenance work orders (MWOs) are an integral part of the maintenance workflow. These documents allow technicians to capture vital aspects of a maintenance job, including observed symptoms, potential causes, and solutions implemented. MWOs have often been disregarded during analysis because of the unstructured nature of the text they contain. However, research efforts have recently emerged that clean MWOs for analysis. One such approach is a tagging method which relies on experts classifying and annotating the words used in the MWOs. This method greatly reduces the volume of words used in the MWOs and links words, including misspellings, that have the same or similar meanings. However, one issue with this approach and with the practical usage of data-annotation tools on the shop-floor more generally is the usage of only one expert annotator at a time. How do we know that the classifications of a single annotator are correct, or if it is, for example, feasible to divide the tagging task among multiple experts? This paper examines the agreement behavior of multiple isolated experts classifying and annotating MWO data, and provides implications for implementing this tagging technique in authentic contexts. The results described here will help improveMWO classification leading to more accurate analysis of MWOs for decision-making support.