ECE 776 – Information theory

---

Description

This is a course on information theory and its applications at graduate level. Topics include basic concepts and definitions of information measure; asymptotic equipartition property and its applications; theory of data compression; definition and theory of channel capacity; rate distortion theory; network information theory. 

“Information theory […], a unifying theory [of data compression and transmission] with profound intersections with Probability, Statistics, Computer Science and other fields, continues to set the stage for the development of communications, data storage and processing, and other information technologies.” [Sergio Verdú, see here].

Prerequisites

Basic knowledge of random signal analysis at the level of ECE 673 is required.

Instructor

Dr. Osvaldo Simeone
Email: osvaldo.simeone@ njit.edu
Phone: (973) 596-5710
Office: ECE 211
Office Hour: Wednesday 4-6pm

Textbook

Elements of Information theory
T. M. Cover and J. A. Thomas
Wiley.

Further reading:

Information theory and reliable communication

R. G. Gallager

John Wiley & Sons, Inc.

---

Requirements

There will one midterm (40%), and one final exam (40%) and a project (20%) to be completed by the date of the final exam. Weekly problems will be assigned but not graded. Some problem sets might involve Matlab simulation. You can obtain a copy of Matlab software from the campus computing facility. 

For the project, a recent paper will be assigned to each student based on individual interests. Twenty-minute presentations to the class on the selected subject will be scheduled by the end of the semester.

---

Some links

• A documentary about C. E. Shannon.
• Information-theoretic inequality prover
• A brief probability refresher from prof. Kschischang (University of Toronto) (PostScript file or a PDF file).
• The original Shannon's paper: C. E. Shannon, ``A mathematical theory of communication,'' Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, July and October, 1948 (as re-typed and corrected by Emre Telatar).
• The significance of Shannon's work, according to Aaron Wyner.
• The IEEE Information Theory Society Home page.
• David MacKay's 1995 short course in Information Theory.
• A list of important references.
• Writing a novel without using the letter “E”.
• Generating real random numbers.
• Shannon, gambling, organized crime, Wall Street… : a light reading.
• The Information: a history, a theory and a flood (J. Gleick).
• A cartoon history of… (L. Gonick)

---

Exams

Spring 2007

· Midterm: solution and grades.

· Projects.

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Fall 2008

· Midterm: solution and grades.

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Spring 2010

· Midterm: solution and grades

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Spring 2011

· Midterm: solution

· Final: solution

Spring 2011

· Midterm: solution

· Final: solution

---

Spring 2012: Tentative schedule

Week	Date	Plan	Chapter covered
1	Jan. 17	Information Measures	2
2	Jan. 24	AEP, Entropy Rate	3, 4
3	Jan. 31	Data Compression	5
4	Feb. 7	Channel Capacity	8
5	Feb. 14	Channel Capacity	8
6	Feb. 21	Channel Capacity	8
7	Feb. 28	Midterm
8	Mar. 6	Differential Entropy	9, 10
9	Mar. 20	Gaussian Channel	9, 10
10	Mar. 27	Gaussian Channel	13
11	Apr. 3	Gaussian Channel, Gambling	13
12	Apr. 10	Rate Distortion Theory	14
13	Apr. 17	Rate Distortion Theory	14
14	Apr. 24	Project presentations
15		Final