Selective comprehension for referring expression by prebuilt entity dictionary with modular networks

Abstract

Referring expression comprehension, known as the technique of localizing entities in an image based on natural language expression, is still a challenging task far from solved. In literature, researchers always focused on how to localize the correct image region according to a natural language expression and never questioned the correctness of the expression. In practical scenarios, the situation is common. For example, there is a pumpkin on the table, but the expression is “there is a watermelon on the table”. It is obvious that incorrect location can be derived from a wrong expression, which state-of-the-art approaches cannot avoid. In this paper, we propose modular networks to solve this problem, which includes three main parts, i.e. the expression filtering module, the expression analysis module and the localization module. Specifically, the expression filtering module adopts an entity dictionary to list all the objects in the image, which is prebuilt by an object detection method, to discriminate whether an expression is correct or not. In this way, our model realizes selective comprehension of referring expression, which can output a “wrong expression” feedback instead of a wrong image region localization when an expression is determined as wrong. Sufficient experiments shows that our model can efficiently filter wrong expressions and effectively solve the problem of referring expression compression in practical scenarios.