Dependency detection in MobiMine and random matrices

Abstract

This paper describes a novel approach to detect correlation from data streams in the context of MobiMine - an experimental mobile data mining system. It presents a brief description of the MobiMine and identifies the problem of detecting dependencies among stocks from incrementally observed financial data streams. This is a non-trivial problem since the stock-market data is inherently noisy and small incremental volumes of data makes the estimation process more vulnerable to noise. This paper presents EDS, a technique to estimate the correlation matrix from data streams by exploiting some properties of the distribution of eigenvalues for random matrices. It separates the "information" from the "noise" by comparing the eigen-spectrum generated from the observed data with that of random matrices. The comparison immediately leads to a decomposition of the covariance matrix into two matrices: one capturing the "noise" and the other capturing useful "information." The paper also presents experimental results using Nasdaq 100 stock data. © 2002 Springer-Verlag Berlin Heidelberg.