A Comprehensive Survey of Knowledge-Based Visual Question Answering Systems: The Lifecycle of Knowledge in Visual Reasoning Task

Abstract

Knowledge-based Visual Question Answering (KB-VQA) extends general Visual Question Answering by requiring external knowledge beyond the provided visual and textual inputs, facilitating more complex real-world applications. KB-VQA introduces unique challenges, including the alignment of heterogeneous information from diverse modalities and sources, the retrieval of relevant knowledge from large-scale and noisy repositories, and the execution of complex reasoning to infer answers from the combined context. With the advancement of large language models, KB-VQA systems have undergone a notable transformation, where LLMs serve as powerful knowledge repositories, retrieval-augmented generators and strong reasoners. Despite substantial progress, there is a lack of a recent, systematic survey that organizes and reviews the evolving landscape of existing KB-VQA methods. This survey aims to fill this gap by establishing a structured taxonomy of KB-VQA approaches and decomposing mainstream systems into three fundamental stages: knowledge representation, knowledge retrieval, and knowledge reasoning. Through an examination of existing techniques employed at each stage, this survey identifies persistent challenges and outlines promising future research directions, providing a foundation for advancing KB-VQA models and their applications.