Zeroth-order stochastic alternating direction method of multipliers for nonconvex nonsmooth optimization

7Citations
Citations of this article
65Readers
Mendeley users who have this article in their library.

Abstract

Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box learning and bandit feedback, ADMM could fail because the explicit gradients of these problems are difficult or even infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values are only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods, they build on the convexity of objective function. Clearly, these existing zeroth-order methods are limited in many applications. In the paper, thus, we propose a class of fast zeroth-order stochastic ADMM methods (i.e., ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems with multiple nonsmooth penalties, based on the coordinate smoothing gradient estimator. Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have convergence rate of O(1/T), where T denotes the number of iterations. In particular, our methods not only reach the best convergence rate of O(1/T) for the nonconvex optimization, but also are able to effectively solve many complex machine learning problems with multiple regularized penalties and constraints. Finally, we conduct the experiments of black-box binary classification and structured adversarial attack on black-box deep neural network to validate the efficiency of our algorithms.

Cite

CITATION STYLE

APA

Huang, F., Gao, S., Chen, S., & Huang, H. (2019). Zeroth-order stochastic alternating direction method of multipliers for nonconvex nonsmooth optimization. In IJCAI International Joint Conference on Artificial Intelligence (Vol. 2019-August, pp. 2549–2555). International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2019/354

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free