Computer Vision Tagging the Metropolitan Museum of Art's Collection: A Comparison of Three Systems

Abstract

Computer vision algorithms are increasingly being applied to museum collections to identify patterns, colors, and subjects by generating tags for each object image. There are multiple off-the-shelf systems that offer an accessible and rapid way to undertake this process. Based on the highlights of the Metropolitan Museum of Art's collection, this article examines the similarities and differences between the tags generated by three well-known computer vision systems (Google Cloud Vision, Amazon Rekognition, and IBM Watson). The results provide insights into the characteristics of these taxonomies in terms of the volume of tags generated for each object, their diversity, typology, and accuracy. In consequence, this article discusses the need for museums to define their own subject tagging strategy and selection criteria of computer vision tools based on their type of collection and tags needed to complement their metadata.