A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering

Abstract

Recent studies have pointed many well-developed Visual Question Answering (VQA) systems suffer from bias problem. Despite the remarkable performance gained on In-Distribution (ID) datasets, the VQA model might capture the superficial correlation from question to answer rather than showing real reasoning abilities. Therefore, when switching to Out-of-Distribution (OOD) dataset, whose test distribution is unknown or even reversed with the training set, significant drops appear. Efforts have been devoted to negative bias brought by language prior but are still limited by two aspects. First, most current debiasing methods achieve promising OOD generalization ability with a sacrifice of the ID performance. Second, they are restricted by exploiting comprehensive biases, since weakening the language bias is mainly focused and few works consider vision bias. In this paper, we investigate a straightforward way to mitigate bias problem for VQA task by subtracting bias score from VQA base score. Then we design two bias learning branches to detect more bias, which is combined with a dynamical constraint loss to alleviate the problem of over-correction and insufficient debiasing. We evaluate our method on the challenging VQA v2.0 and VQA-CP V2.0 datasets and achieve significant improvement.