QACE: Asking Questions to Evaluate an Image Caption

Abstract

In this paper, we propose QACE, a new metric based on Question Answering for Caption Evaluation. QACE generates questions on the evaluated caption and checks its content by asking the questions on either the reference caption or the source image. We first develop QACERef that compares the answers of the evaluated caption to its reference, and report competitive results with the state-of-the-art metrics. To go further, we propose QACEImg, which asks the questions directly on the image, instead of reference. A Visual-QA system is necessary for QACEImg. Unfortunately, the standard VQA models are framed as a classification among only a few thousand categories. Instead, we propose Visual-T5, an abstractive VQA system. The resulting metric, QACEImg is multi-modal, reference-less, and explainable. Our experiments show that QACEImg compares favorably w.r.t. other reference-less metrics. We will release the pre-trained models to compute QACE.