Demo: WhisperFlow: speech foundation models in real time

Abstract

Speech foundation models, such as OpenAI's Whisper, become the state of the art in speech understanding due to their strong accuracy and generalizability. Yet, their applications are mostly limited to processing pre-recorded speech, whereas processing of streaming speech, in particular doing it efficiently, remains rudimentary.We present a novel framework, WhisperFlow, which embodies both model and system optimizations. (1) Hush word as a short, learnable audio segment; appended to a voice input, a hush word gracefully stops the speech model from processing more input without hallucination; (2) Beam pruning, which aligns streaming audio buffers over time and reuses results from earlier decoding rounds, therefore significantly accelerating decoding; and (3) CPU/GPU pipelining, which not only maps to the encoding/decoding stages dynamically, but also tunes to an optimal resource ratio, respecting the encoding/decoding speed that varies across voice inputs, models, and hardware.We demonstrate WhisperFlow on a Macbook pro with M4 pro SoC with 14 CPU cores and 20 GPU cores on real-world conversation transcription tasks. The WhisperFlow delivers high fidelity transcripts with the Whisper medium model, also maintains the per-word latency within 1 second.