Acceleration by stepsize hedging: Silver Stepsize Schedule for smooth convex optimization

Abstract

We provide a concise, self-contained proof that the Silver Stepsize Schedule proposed in our companion paper directly applies to smooth (non-strongly) convex optimization. Specifically, we show that with these stepsizes, gradient descent computes an ε-minimizer in O(ε-logρ2)=O(ε-0.7864) iterations, where ρ=1+2 is the silver ratio. This is intermediate between the textbook unaccelerated rate O(ε-1) and the accelerated rate O(ε-1/2) due to Nesterov in 1983. The Silver Stepsize Schedule is a simple explicit fractal: the i-th stepsize is 1+ρν(i)-1 where ν(i) is the 2-adic valuation of i. The design and analysis are conceptually identical to the strongly convex setting in our companion paper, but simplify remarkably in this specific setting.