ECE 776 – Information theory

---

Description

This is a course on information theory and its applications at graduate level. Topics include basic concepts and definitions of information measure; asymptotic equipartition property and its applications; theory of data compression; definition and theory of channel capacity; rate distortion theory; network information theory. 

“Information theory […], a unifying theory [of data compression and transmission] with profound intersections with Probability, Statistics, Computer Science and other fields, continues to set the stage for the development of communications, data storage and processing, and other information technologies.” [Sergio Verdú, see here].

Prerequisites

Basic knowledge of random signal analysis at the level of ECE 673 is required.

Instructor

Dr. Osvaldo Simeone
Email: osvaldo.simeone@ njit.edu
Phone: (973) 596-5710
Office: ECE 211
Office Hour: Wednesday 4-6pm

Textbook

Elements of Information theory
T. M. Cover and J. A. Thomas
Wiley.

Further reading:

Information theory and reliable communication

R. G. Gallager

John Wiley & Sons, Inc.

---

Requirements

There will one midterm (40%), and one final exam (40%) and a project (20%) to be completed by the date of the final exam. Weekly problems will be assigned but not graded. Some problem sets might involve Matlab simulation. You can obtain a copy of Matlab software from the campus computing facility. 

For the project, a recent paper will be assigned to each student based on individual interests. Twenty-minute presentations to the class on the selected subject will be scheduled by the end of the semester.

---

Some links

• What is information theory?
• A documentary about C. E. Shannon.
• The pre-history of information theory
• What is information theory? (S. Verdú)
• Shannon's original paper: C. E. Shannon, ``A mathematical theory of communication,'' Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, July and October, 1948 (as re-typed and corrected by Emre Telatar).
• The significance of Shannon's work, according to Aaron Wyner.
• Shannon, gambling, organized crime, Wall Street… : a light reading.
• The Information: a history, a theory and a flood (J. Gleick).
• A cartoon history of… (L. Gonick)
• The IEEE Information Theory Society Home page.
• A list of important references.
• Writing a novel without using the letter “E”.
• Generating real random numbers.
• A brief probability refresher from prof. Kschischang (University of Toronto) (PostScript file or a PDF file).
• David MacKay's 1995 short course in Information Theory.
• Information-theoretic inequality prover

---

Exams

Spring 2007

· Midterm: solution and grades.

· Projects.

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Fall 2008

· Midterm: solution and grades.

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Spring 2010

· Midterm: solution and grades

· Final: solution and grades (6-7: C+, 7-8: B, 8-9: B+, 9-10: A).

Spring 2011

· Midterm: solution

· Final: solution

Spring 2012

· Midterm: solution

· Final: solution

Spring 2013

· Midterm: text and solution

· Final: text and solution

Spring 2014

· Midterm: text and solution

· Final: text and solution

Spring 2015

· Midterm: text and solution

· Final: text and solution

---

Spring 2016: Tentative schedule

Week	Date	Plan	Chapter covered
1	Jan. 25	Introduction	1
2	Feb. 1	Information Measures	2
3	Feb. 8	Information Measures	2
4	Feb. 15	Zero-Error Data Compression	5
5	Feb. 22	AEP	3
6	Feb. 29	Lossless Data Compression	3
7	Mar. 7	Midterm
8	Mar. 21	Entropy Rate	4
9	Mar. 28	Channel Capacity	7
10	Apr. 4	Channel Capacity	7
11	Apr. 11	Differential entropy	8
12	Apr. 18	Gaussian channel	9
13	Apr. 25	Rate-distortion theory	10
14	May 2	Rate-distortion theory	10
15		Final