Storybranch - generating multimedia content from novels

Abstract

We present STORYBRANCH - an automated system for generating multimedia content from long texts such as novels and fanfiction. The STORYBRANCH pipeline includes structured information extraction, text parsing and processing, content generation using Gen-AI models and syncronization of different streams (audio, video, background). Our system is highly modular and can efficiently generate three different types of multimodal content: audiobooks, simple animated videos, and visual novel text-and-image-style video games. STORYBRANCH successfully addresses challenges such as generating unique and consistent image and voice for each character and narrator, identifying and generating background images and sounds effects, and syncronizing character expressions and lip movement with text. As part of the STORYBRANCH, we develop and release BookNLP2 - a new open-source library for parsing and extracting information from books, based on the legacy library BookNLP.