A Survey of Emerging Trend Detection in Textual Data Mining

Abstract

In this c hapter w e describe sev eral systems that detect emerg? ing trends in textual data? Some of the systems are semi?automatic? requiring user input to begin processing? others are fully?automatic? producing output from the input corpus without guidance? F or eac h Emerging T rend Detection ?ETD? system w e describe componen ts including linguistic and statistical features? learning algorithms? training and test set generation? visualization and ev aluation? W e also pro vide a brief o erview of sev v eral commercial products with capabilities for detecting trends in textual data? follo ed b w y an in? dustrial viewpoin t describing the importance of trend detection tools? and an o erview of ho v w suc h tools are used? This review of the literature indicates that m h progress has uc been made to ard automating the process of detecting emerging w trends? but there is room for impro emen v t? All of the projects w e review ed rely on a h uman domain expert to separate the emerging trends fromnoiseinthe system? F urthermore? w edisco ered that few v projects ha e used formal ev v aluation methodologies to determine the e?ectiv eness of the systems being created? Dev elopmen t and use of e?ectiv e metrics for ev aluation of ETD systems is critical? W ork con tin ues on the semi?automatic and fully?automatic sys? tems w e are dev eloping at Lehigh Univ ersit y ?HDD ?? In addition to adding formal ev aluation componen ts to our systems? w e are also re? searc hing methods for automatically dev eloping training sets and for merging mac hine learning and visualization to dev elop more e?ectiv e ETD applications?