Online Learning in the Embedded Manifold of Low-rank Matrices

  • Shalit U
  • Weinshall DAPHNA D
  • Chechik G
  • 101


    Mendeley users who have this article in their library.
  • 25


    Citations of this article.


When learning models that are represented in matrix forms, enforcing a low-rank constraint can dramatically improve the memory and run time complexity, while providing a natural regularization of the model. However, naive approaches to minimizing functions over the set of low-rank matrices are either prohibitively time consuming (repeated singular value decomposition of the matrix) or numerically unstable (optimizing a factored representation of the low-rank matrix). We build on recent advances in optimization over manifolds, and describe an iterative online learning procedure, consisting of a gradient step, followed by a second-order retraction back to the manifold. While the ideal retraction is costly to compute, and so is the projection operator that approximates it, we describe another retraction that can be computed efficiently. It has run time and memory complexity of O ((n + m)k) for a rank-k matrix of dimension m × n, when using an online procedure with rank-one gradients. We use this algorithm, LORETA, to learn a matrix-form similarity measure over pairs of documents represented as high dimensional vectors. LORETA improves the mean average precision over a passive-aggressive approach in a factorized model, and also improves over a full model trained on pre-selected features using the same memory requirements. We further adapt LORETA to learn positive semi-definite low-rank matrices, providing an online algorithm for low-rank metric learning. LORETA also shows consistent improvement over standard weakly supervised methods in a large (1600 classes and 1 million images, using ImageNet) multi-label image classification task.

Author-supplied keywords

  • Riemannian manifolds
  • low rank
  • metric learning
  • multitask learning
  • online learning
  • retractions

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

  • PMID: WOS:000303046000007
  • PUI: 364381958
  • ISBN: 1532-4435
  • ISSN: 15324435
  • SGR: 84857807897
  • SCOPUS: 2-s2.0-84857807897


  • Uri Shalit

  • Daphna Weinshall DAPHNA

  • Gal Chechik

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free