Stalker, a multilingual text mining search engine for open source intelligence

Abstract

Open Source Intelligence (OSINT) is an intelligence gathering discipline that involves collecting information from open sources and analyzing it to produce usable intelligence. The international Intelligence Communities have seen open sources grow increasingly easier and cheaper to acquire in recent years. But up to 80% of electronic data is textual and most valuable information is often hidden and encoded in pages which are neither structured, nor classified. The process of accessing all these raw data, heterogeneous in terms of source and language, and transforming them into information is therefore strongly linked to automatic textual analysis and synthesis, which are greatly related to the ability to master the problems of multilinguality. This paper describes a content enabling system that provides deep semantic search and information access to large quantities of distributed multimedia data for both experts and general public. STALKER provides with a language independent search and dynamic classification features for a broad range of data collected from several sources in a number of culturally diverse languages. © 2009 Springer-Verlag Berlin Heidelberg.