Ground Truth Data, Content, Metrics, and Analysis

Abstract

Buy the truth and do not sell it. —Proverbs 23:23 This chapter discusses several topics pertaining to ground truth data , the basis for computer vision metric analysis. We look at examples to illustrate the importance of ground truth data design and use, including manual and automated methods. We then propose a method and corresponding ground truth dataset for measuring interest point detector response as compared to human visual system response and human expectations. Also included here are example applications of the general robustness criteria and the general vision taxonomy developed in Chapter 5 as applied to the preparation of hypothetical ground truth data. Lastly, we look at the current state of the art, its best practices, and a survey of available ground truth datasets. Key topics include: Creating and collecting ground truth data: manual vs. synthetic • methods Labeling and describing ground truth data: automated vs. human • annotated Selected ground truth datasets • Metrics paired with ground truth data • Over-fitting, under-fitting, and measuring quality • Publically available datasets • An example scenario that compares the human visual system to • machine vision detectors, using a synthetic ground truth dataset Ground truth data may not be a cutting-edge research area, however it is as important as the algorithms for machine vision. Let's explore some of the best-known methods and consider some open questions. In the context of computer vision, ground truth data includes a set of images, and a set of labels on the images, and defining a modelfor object recognition as discussed in Chapter 4, including the count, location, and relationships of key features. The labels are added either by a human or automatically by image analysis, depending on the complexity of the problem. The collection of labels, such as interest points, corners, feature descriptors, shapes, and histograms, form a model. A model may be trained using a variety of machine learning methods. At run-time, the detected features are fed into a classifier to measure the correspondence between detected features and modeled features. Modeling, classification, and training are statistical and machine learning problems , however, that are outside the scope of this book. Instead, we are concerned here with the content and design of the ground truth images. Creating a ground truth dataset, then, may include condieration of the following major tasks: • Model design. The model defines the composition of the objects—for example, the count, strength, and location relationship of a set of SIFT features. The model should be correctly fitted to the problem and image data so as to yield meaningful results.