Automatic curation system using multimodal analysis approach (MAA)

Abstract

Extracting sports highlights and summarizing the most exciting moments of the match is an important task for the broadcast media. However, it requires intensive video editing. We propose a Multimodal Analysis Approach (MAA) for auto-curating sports highlights, and use it to create a real-world system for editing aids of soccer highlight reels. MAA fuses information from the players actions (action recognition), the landmarks (image classification), the scores on the scoreboard (OCR) and the commentators tone of the voice (audio detection) to determine the most exciting moments of a match. In addition, we use the face recognition technology to make star highlights. The shot-boundary detection method is developed to accurately identify the start and end frames of highlights for content summaries. The proposed system has performed real-time highlights extraction from the video stream of the FIFA world cup 2018. Moreover, MAA produces highlights with better quality by comprehensive user studies from multiple participants subjects.