CS6800 15F: Syllabus

Note: This is an approximate syllabus; it may change at any time.

1. Lecture 01, Fri, Sep 11 2015
   • Admistrivia
   • Introduction to Information Theory and its applications

2. Lecture 02, Tue, Sep 15 2015
   • Probability primer I

3. Lecture 03, Fri, Sep 18 2015
   • Probability primer II
   • Homework 01 assigned

4. Lecture 04, Tue, Sep 22 2015
   • Entropy and its properties
   • Reading: Cover and Thomas 2.1-2

5. Lecture 05, Fri, Sep 25 2015
   • Conditional entropy, relative entropy, mutual information
   • Reading: Cover and Thomas 2.2-4
   • Homework 01 due
   • Homework 02 assigned

6. Lecture 06, Tue, Sep 29 2015
   • Chain rules, data processing inequality, Fano's inequality
   • Reading: Cover and Thomas 2.5-6, 2.8, 2.10

7. Lecture 07, Fri, Oct 02 2015
   • Compression I: codes and decodability, Kraft's inequality, bounds on optimal codes
   • Reading: Cover and Thomas 5.1-4
   • Homework 02 due
   • Homework 03 assigned

8. Lecture 08, Tue, Oct 06 2015
   • Compression II: more Kraft's inequality, block coding, Huffman codes
   • Reading: Cover and Thomas 5.5-8

9. Lecture 09, Fri, Oct 09 2015
   • Compression III: twenty questions, randomness
   • Reading: Cover and Thomas 5.7, 5.10, 5.11, 13.3
   • Homework 03 due

10. Lecture 10, Tue, Oct 13 2015
    • Markov chains, entropy rate of stochastic processes
    • Reading: Cover and Thomas 4

11. Lecture 11, Fri, Oct 16 2015
    • Asymptotic Equipartition Property (AEP) and its consequences
    • Reading: Cover and Thomas 3
    • Homework 04 assigned

12. Lecture 12, Tue, Oct 20 2015
    • Lempel-Ziv, universal source coding
    • Reading: Cover and Thomas 13.4-5

13. Lecture 13, Fri, Oct 23 2015
    • Arithmetic codes, Fibonacci codes, Elias Omega codes
    • Start information theory and statistics: the method of types and applications
    • Reading: Cover and Thomas 13.2-3, 11.1-6

14. Lecture 14, Tue, Oct 27 2015
    • Finish information theory and statistics: the method of types and applications
    • Start large deviation theory
    • Reading: Cover and Thomas 11.1-6
    • Homework 04 due
    • Homework 05 assigned

15. Lecture 15, Fri, Oct 30 2015
    • Finish large deviation theory
    • Start the maximum entropy method
    • Reading: Cover and Thomas 12.1-2

16. Lecture 16, Tue, Nov 03 2015
    • Finish the maximum entropy method
    • Reading: Cover and Thomas 12.1-2

17. Remaining lectures to be announced...

Switch to:

• CS6800 15F Home Page