MASTER SAR

SUPELEC – ENS CACHAN – UNIVERSITY PARIS–SUD 11

Name of UE: Elements of Information Theory
Professor in charge: Olivier RIOUL
Scheduled volume: 24H / 2.5 ECTS

Reference
SAR–C1
Information Theory
Lecturers: O. RIOUL, P. PIANTANIDA

Elements of Information Theory

Instructor: Prof. Olivier RIOUL
ENST – Department COMELEC
Office: A304
Phone: +33 (0)1 45 81 78 45
Fax: +33 (0)1 45 81 00 20
E–mail: rioul@enst.fr
URL: http://comelec.enst.fr/~rioul/

Instructor: Dr. Pablo PIANTANIDA
SUPELEC – Department of Telecommunications
Office: A4-19
Phone: +33 (0)1 69 85 14 50
Fax: +33 (0)1 69 85 14 69
E–mail: pablo.piantanida@supelec.fr
URL: http://www.supelec.fr/ecole/radio/piantanida.html/

Time: S3 / 24H CM
Location: SUPELEC – Department of Telecommunications
Reception hours: After class or by setting an appointment (e-mail)
Pre–requisites: SAR-B1
Grading: Final exam
Homework: Theoretical exercises and/or programming exercises

Abstract
This is a graduate-level introduction to the fundamental ideas and results of information
theory. The course moves quickly but does not assume prior study in information theory.
It is intended for graduate students from mathematics, engineering or related areas
wanting a good background in fundamental and applicable information theory. It also
provides solid preparation for advanced courses in information theory and for various
courses in the field of communications.
Roughly the first third of the course discusses elementary measures and properties of
information at a more sophisticated level. The middle third discusses method of Types
and the main results of Shannon’s theory, manly the coding theorems for source and
channel coding. The remainder touches on topics that are explored more fully in later
courses, converses, capacity of wireless channels, etc.

Course outline

Lecture 1. Properties of Shannon's Information Measures [1, chap. 2,11][2, chap. 1,2,3]

Introduction and Main Definitions

Entropy, Relative Entropy and Mutual Information

Venn Diagrams

Jensen's Inequality and Properties of Relative Entropy

Differential and Absolute Entropy, Entropy of Binary and Gaussian RVs

Lecture 2. Markov Chain, Fundamental Inequalities and Entropy of Stationary Sources [1,
chap. 2, 16][2, chap. 4,5]

Convexity Properties of Information Measures

Entropy Maximization

Fano's Inequalities

Markov Chains and Data Processing Inequality

Chain Rules for Entropy, Relative Entropy and Mutual Information

The Log-Sum Inequality and Entropy Power Inequality (EPI)

Lecture 3. Lossless Source Coding [1, chap. 3,4,5][2, chap. 9,12,13][3, chap. 1]

Weak Typical Sequences

Noiseless Source Coding and its Coding Theore

Entropy Rate of Stationary Sources

Proof of the Coding Theorem (DMS)

Lecture 4. Method of Types [3, chap. 2]

Definitions, Type Counting Lemma

Continuity of the Entropy Function

Strong Typical Sets, Delta Sequences

Bounds on the Size of Strong Typical Sets

Lecture 5. Coding Theorem for Noisy Channels [1, chap. 8,10][2, chap. 9-13][4, chap. 3]

Coding Theorem

Capacity of Binary Symmetric Channels, Gaussian Channels (AWGN), etc.

Proof of the Coding Theorem

Fundamental Lemma (Feinstein's Lemma)

Lecture 6. Coding Theorem for Lossy Source Coding [1, chap. 13][2, chap. 10,11,13]

Quantization and Distortion

Coding Theorem

Rate Distortion Function of Binary and Gaussian Sources

Proof of the Coding Theorem

Lecture 7. Converse to the Coding Theorems for Discrete Memoryless Sources and
Channels [1, chap. 8,13][2, chap. 8]

Converse to the Coding Theorem for Discrete Memoryless Channels (DMCs)

Converse to the Coding Theorem for DMCs with Feedback

Converse to the Coding Theorem for the Discrete Memoryless Source (DMSs)

Converse to the Joint Source-Channel Coding for DMSs and DMCs

References
[1] COVER, T.M., and THOMAS, J.A., Elements of Information Theory, Wiley, 1991.
[2] RIOUL, O., Théorie de l’Information et du Codage, Hermès Science – Lavoisier, 2007.
[3] CSISZAR, I., and KORNER, J., Information Theory: Coding Theorems for Discrete
Memoryless Systems, Academic Press, 1997.
[4] ASH, R.B., Information Theory, Interscience Publishers, 1966.
[5] GOLDSMITH, A., Wireless Communications, Cambridge University Press, 2005.
[6] TSE, D., VISWANATH, P., Fundamentals of Wireless Communications, Cambridge
University Press, 2005.
[7] GALLAGER, R.G., Information and Reliable Communications, Wiley, 1968.
[8] YEUNG, R.W., A First Course in Information Theory, Kluwer Academic/Plenum
Publishers, 2002.
[9] MACKAY, D.J.C., Information Theory, Inference, and Learning Algorithms, Cambridge
University Press, 2003.

[10] SHANNON, C.E., A mathematical theory of communication,’ Bell Syst. Tech. J., vol.
27, pp. 623-656, Oct. 1948.
[11] SHANNON, C.E., Coding theorems for a discrete source with a fidelity criterion, IRE
Nat. Conv. Rec., part 4, pp. 142-163, 1959.