Rapidly developing NLP applications for content curation

Abstract

Time and again we are faced, in a number of collaborative research projects, with the challenge of interconnecting various language processing tools to implement certain industry-driven use cases focusing, for the most part, upon digital content curation processes. In this paper we first describe several of the relevant projects and their technology platforms, followed by a description of the corresponding use cases and their requirements. The content curation platform we focus upon in this article and which has been implemented as a prototype makes use of a large number of NLP services, which we also build upon for other use cases and prototypes. In addition to the implemented NLP services, the article presents a workflow manager for the flexible creation and customisation of processing workflows that make use of the above mentioned NLP services. Based on the four key principles of generality, flexibility, scalability and efficiency, we present the first version of the workflow manager by providing details on its custom definition language, explaining the communication components and the general system architecture and setup. The paper also addresses challenges in interoperability across different NLP tasks and hardware-based resource use.