Dimensionality reduction: Beyond the Johnson-Lindenstrauss bound

Abstract

Dimension reduction of metric data has become a useful technique with numerous applications. The celebrated Johnson-Lindenstrauss lemma states that any n-point subset of Euclidean space can be embedded in O(ε -2log n)- dimension with (1 + ε)-distortion. This bound is known to be nearly tight. In many applications the demand that all distances should be nearly preserved is too strong. In this paper we show that indeed under natural relaxations of the goal of the embedding, an improved dimension reduction is possible where the target dimension is independent of n. Our main result can be viewed as a local dimension reduction. There are a variety of empirical situations in which small distances are meaningful and reliable, but larger ones are not. Such situations arise in source coding, image processing, computational biology, and other applications, and are the motivation for widely-used heuristics such as Isomap and Locally Linear Embedding. Pursuing a line of work begun by Whitney, Nash showed that every C1 manifold of dimension d can be embedded in ℝ2d+2 in such a manner that the local structure at each point is preserved isometrically. Our work is an analog of Nash's for discrete subsets of Euclidean space. For perfect preservation of infinitesimal neighborhoods we substitute near-isometric embedding of neighborhoods of bounded cardinality. We show that any finite subset of Euclidean space can be embedded in O(ε-2 log k)-dimension while preserving with (1 + ε)-distortion the distances within a "core neighborhood" of each point. (The core neighborhood is a metric ball around the point, whose radius is a substantial fraction of the radius of the ball of cardinality k, the k-neighborhood.) When the metric space satisfies a weak growth rate property, the guarantee applies to the entire k-neighborhood (with some dependency of the embedding dimension on the growth rate). We also show how to obtain a global embedding that also keeps distant points well-separated (at the cost of dependency on the doubling dimension of the space). As an application of our methods we obtain an (Assouad-style) dimension reduction for finite subsets of Euclidean space where the metric is raised to some fractional power (the resulting metrics are known as snowflakes). We show that any such metric X can be embedded in dimension Ō(ε-3 dim(X)) with 1 + ε distortion, where dim(X) is the doubling dimension, a measure of the intrinsic dimension of the set. This result improves recent work by Gottlieb and Krauthgamer [20] to a nearly tight bound. The new dimension reduction results are useful for such as clustering and distance labeling.