Brain relevance feedback for interactive image generation

Abstract

Brain-computer interfaces (BCIs) are increasingly used to perform simple operations such as a moving a cursor, but have remained of limited use for more complex tasks. In our new approach to BCI, we use brain relevance feedback to control a generative adversarial network (GAN). We obtained EEG data from 31 participants who viewed face images while concentrating on particular facial features. Following, an EEG relevance classifier was trained and propagated as feedback on the latent image representation provided by the GAN. Estimates for individual vectors matching the relevant criteria were iteratively updated to optimize an image generation process towards mental targets. A double-blind evaluation showed high performance (86.26% accuracy) against random feedback (18.71%), and not significantly lower than explicit feedback (93.30%). Furthermore, we show the feasibility of the method with simultaneous task targets demonstrating BCI operation beyond individual task constraints. Thus, brain relevance feedback can validly control a generative model, overcoming a critical limitation of current BCI approaches.