A neural, interactive-predictive system for multimodal sequence to sequence tasks

Abstract

We present a demonstration of a neural interactive-predictive system for tackling mul- timodal sequence to sequence tasks. The sys- tem generates text predictions to different se- quence to sequence tasks: machine translation, image and video captioning. These predictions are revised by a human agent, who introduces corrections in the form of characters. The sys- tem reacts to each correction, providing alter- native hypotheses, compelling with the feed- back provided by the user. The final objective is to reduce the human effort required during this correction process. This system is implemented following a client-server architecture. For accessing the system, we developed a website, which com- municates with the neural model, hosted in a local server. From this website, the dif- ferent tasks can be tackled following the interactive-predictive framework. We open- source all the code developed for building this system. The demonstration in hosted in http://casmacat.prhlt.upv.es/ interactive-seq2seq.