A Tutorial on Convex Optimization
Haitham Hindi
Palo Alto Research Center (PARC), Palo Alto, California
email: hhindi@parc.com
Abstract— In recent years, convex optimization has be-
come a computational tool of central importance in engi-
neering, thanks to it’s ability to solve very large, practical
engineering problems reliably and efficiently. The goal of
this tutorial is to give an overview of the basic concepts of
convex sets, functions and convex optimization problems, so
that the reader can more readily recognize and formulate
engineering problems using modern convex optimization.
This tutorial coincides with the publication of the new book
on convex optimization, by Boyd and Vandenberghe [7],
who have made available a large amount of free course
material and links to freely available code. These can be
downloaded and used immediately by the audience both
for self-study and to solve real problems.
I. INTRODUCTION
Convex optimization can be described as a fusion
of three disciplines: optimization [22], [20], [1], [3],
[4], convex analysis [19], [24], [27], [16], [13], and
numerical computation [26], [12], [10], [17]. It has
recently become a tool of central importance in engi-
neering, enabling the solution of very large, practical
engineering problems reliably and efficiently. In some
sense, convex optimization is providing new indispens-
able computational tools today, which naturally extend
our ability to solve problems such as least squares and
linear programming to a much larger and richer class of
problems.
Our ability to solve these new types of problems
comes from recent breakthroughs in algorithms for solv-
ing convex optimization problems [18], [23], [29], [30],
coupled with the dramatic improvements in computing
power, both of which have happened only in the past
decade or so. Today, new applications of convex op-
timization are constantly being reported from almost
every area of engineering, including: control, signal
processing, networks, circuit design, communication, in-
formation theory, computer science, operations research,
economics, statistics, structural design. See [7], [2], [5],
[6], [9], [11], [15], [8], [21], [14], [28] and the references
therein.
The objectives of this tutorial are:
1) to show that there are straight forward, systematic
rules and facts, which when mastered, allow one to
quickly deduce the convexity (and hence tractabil-
ity) of many problems, often by inspection;
2) to review and introduce some canonical opti-
mization problems, which can be used to model
problems and for which reliable optimization code
can be readily obtained;
3) to emphasize modeling and formulation; we do
not discuss topics like duality or writing custom
codes.
We assume that the reader has a working knowledge of
linear algebra and vector calculus, and some (minimal)
exposure to optimization.
Our presentation is quite informal. Rather than pro-
vide details for all the facts and claims presented, our
goal is instead to give the reader a flavor for what is
possible with convex optimization. Complete details can
be found in [7], from which all the material presented
here is taken. Thus we encourage the reader to skip
sections that might not seem clear and continue reading;
the topics are not all interdependent.
Also, in order keep the paper quite general, we
have tried to not to bias our presentation toward any
particular audience. Hence, the examples used in the
paper are simple and intended merely to clarify the
optimization ideas and concepts. For detailed examples
and applications, the reader is refered to [7], [2], and
the references therein.
We now briefly outline the paper. Sections II and III,
respectively, describe convex sets and convex functions
along with their calculus and properties. In section IV,
we define convex optimization problems, at a rather
abstract level, and we describe their general form and
desirable properties. Section V presents some specific
canonical optimization problems which have been found
to be extremely useful in practice, and for which effi-
cient codes are freely available. Section VI comments
briefly on the use of convex optimization for solving
nonstandard or nonconvex problems. Finally, section VII
concludes the paper.
Motivation
A vast number of design problems in engineering can
be posed as constrained optimization problems, of the