EE667A - Information Theory

Introduction: Entropy, Relative Entropy, Mutual Information Inequalities, Entropy rate. Asymptotic Equipartition Property (AEP): Consequences of the AEP, Typical Sequences, Shannon McMillan Breiman Theorem. Data Compression: Block to variable length codes, Shannon Fano code, Huffman code, variable to fixed length coding Tunstal code, variable to variable length codes/ arithmetic code. Channel capacity: Discrete Memory less Channel, Joint Typicality, Channel Coding Theorem and its converse, Feedback capacity, Source Channel Separation Theorem. Differential Entropy: Definition, Properties. Gaussian Channel: Definition, Parallel Gaussian Channels, Channels with Colored Gaussian Noise, Gaussian Channels with Feedback. Rate Distortion Theory: Rate Distortion Function, Rate Distortion theorem and its converse, Blahut Arimoto Algorithm. Universal Source Coding: Universal codes, Lempel-Ziv codes; LZ 78, LZW, Sliding Window Lempel Ziv algorithm (LZ77). Network Information Theory: Gaussian Multi User Channels, Multiple Access Channel, Broadest Channel, Encoding of Correlated Sources.