Affective Image Captioning for Visual Artworks Using Emotion-Based Cross-Attention Mechanisms

Abstract

Within the museum community, the automatic generation of artwork description is expected to accelerate the improvement of accessibility for visually impaired visitors. Captions that describe artworks should be based on emotions because art is inseparable from viewers' emotional reactions. By contrast, artworks typically do not have unique interpretations; thus, it is difficult for systems to reflect the specified emotions in captions precisely. Most existing methods attempt to leverage predicted emotion labels from images to generate emotion-oriented captions; however, they do not allow users to specify arbitrary emotions. In this paper, we aim to build a model that generates emotion-conditioned captions that describe visual art. We propose an affective visual encoder, which integrates emotion attributes and cross-modal joint features of images into visual information over all encoder blocks. Moreover, we introduce affective tokens that fuse grid- and region-based image features to cover both contextual and object-level information. We validated our method on the ArtEmis dataset, and the results demonstrated that our method outperformed baseline methods on all metrics in the emotion-conditioned task.