Mobile visual search for digital heritage applications

Abstract

In this chapter, we demonstrate a complete pipeline for multimedia retrieval on a mobile device. We target the use case of a tourist at a heritage site, who wishes guide herself by clicking an image of an interesting structure to get information about the same. This requires efficient mobile-based instance retrieval techniques over a dataset of 1000s of images. Such a task on mobile requires a significant reduction in the visual index size. To achieve this, we describe a set of strategies that can reduce the size of the visual index structure compared to a standard instance retrieval implementation found on desktops or servers. While our proposed reduction steps affect the overall mean Average Precision (mAP), they are able to maintain a good Precision for the top K results (P K ). We argue that for such offline application, maintaining a good P K is sufficient. Such an instance retrieval framework depends on a well-annotated dataset of images to retrieve from. Photos from tourist and heritage sites can often be described with detailed and part-wise annotations. Manually, annotating a large community photo collection is a costly and redundant process as similar images share the same annotations. Hence, we also demonstrate an interactive web-based annotation tool that allows multiple users to add, view, edit and suggest rich annotations for images in community photo collections. Since, distinct annotations could be few, we have an easy and efficient batch annotation approach using an image similarity graph, pre-computed with instance retrieval and matching. This helps in seamlessly propagating annotations of the same objects or similar images across the entire dataset.