Square peg, round hole: A case study on using Visual Question & Answering in Games

Abstract

The discussion about what can Artificial Intelligence (AI) contribute to games has been running for a long time, however, recent advances in AI show promise of providing new kind of experiences for players and new tools for game developers. In contrast with the traditional Finite State Machine for interaction and response, we consider the scenario of Visual Question & Answering (VQA) - the automatic answering of a textual question about an image. VQA is a tool that can enrich possible answers by combining both visual and textual information. It is also trivial to extrapolate to a game setting without going away from the training domain. In this Work In Progress, we present two original prototypes designed to explore the potential of VQA in games and discuss preliminary findings originated through a Wizard of Oz (WOz) pilot study using VQA to investigate how people interact with such an AI algorithm.