Of gods and goats: Weakly supervised learning of figurative art

Abstract

The objective of this paper is to automatically annotate images of gods and animals in decorations on classical Greek vases. Such images often require expert knowledge in labelling. We start from a large dataset of images of vases with associated brief text descriptions. We develop a weakly supervised learning approach to solve the correspondence problem between the descriptions and unknown image regions. The approach progressively strengthens the supervision so that eventually a Deformable Part Model (DPM) sliding window detector can be learnt (for each god/animal) and used to annotate all vases by detection. There are two key steps: first, text mining the vase descriptions to obtain clusters for each god where there is a visual consistency between at least a subset of the images; and second, discriminatively searching within these clusters for consistent regions that are used as positive training examples for the DPM. The method successfully annotates a large variety of Gods and other animals, and we include a quantitative evaluation over hundreds of images.