Information Theory



This page contains resources about Information Theory in general.

More specific information is included in each subfield.

Subfields and Concepts
See Category:Information Theory for some of  its subfields.
 * Shannon entropy / Information entropy
 * Cross entropy / Joint entropy
 * Conditional entropy
 * Differential entropy
 * Information content
 * Mutual Information
 * Relative entropy / Kullback-Leibler divergence / Information gain
 * Perplexity
 * Entropy encoding
 * Huffman coding
 * Arithmetic coding
 * Algorithmic Information Theory
 * Kolmogorov Complexity / Algorithmic Complexity
 * Rademacher Complexity
 * Algorithmic Probability / Solomonoff Probability
 * Universal Search (by Levin)
 * Algorithmic Randomness (by Martin-Lof)
 * Solomonoff's Theory of Inductive Inference
 * Epicurus' Principle of Multiple Explanations
 * Occam's Razor
 * Bayes' rule
 * Universality probability
 * Universal Turning Machine
 * Minimum Description Length (MDL) principle
 * Minimum Message Length (MML)
 * Algorithmic Statistics
 * Principle of Maximum Entropy
 * Hamming distance
 * Hamming code
 * Wavelets
 * Information bottleneck
 * Neural Network Compression / Model Compression
 * Nodes pruning
 * Weight pruning / Connection pruning
 * Quantization of weights
 * Deep Compression
 * Dynamic Network Surgery
 * SqueezeNet Architecture
 * Structured Sparsity Learning
 * Soft-weight sharing
 * Bayesian Compression
 * Variational Dropout
 * Coding Theory
 * Data Compression / Source Coding
 * Lossy Compression
 * Lossless Compression
 * Probabilistic Data Compression
 * Prediction by partial matching (PPM)
 * Sequence Memoizer
 * Bayesian Networks
 * Shannon's Source Coding Theorem / Noiseless Coding Theorem
 * Error Correction / Channel Coding
 * Cryptographic Coding
 * Shannon–Hartley Theorem
 * Noisy-Channel Coding Theorem
 * Shannon Limit / Shannon Capacity
 * Applications
 * Cryptography
 * Communication Systems
 * Machine Learning / Pattern Recognition
 * Statistical Learning Theory
 * Estimation Theory / Statistical Signal Processing
 * Bayesian Inference

Video Lectures

 * Information Theory and Coding by S.N.Merchant
 * Information Theory, Pattern Recognition, and Neural Networks by David MacKay
 * Information Theory by Raymond W. Yeung
 * Probability, Information Theory and Bayesian Inference by Joaquin Quiñonero Candela
 * Information, Entropy and Computation by Paul Penfield and Seth Lloyd (Notes)

Lecture Notes

 * Information Theory by Tsachy Weissman
 * Information Theory by Muriel Médard
 * Information Theory by Yao Xie
 * Advanced Topics in Information Theory by Stefan M. Moser
 * A Short Course in Information Theory by David J.C. MacKay
 * Information Theory by Radford Neal
 * Information theory in computer science by Mark Braverman
 * Information Theory in Computer Science by Anup Rao
 * Information Theory and its applications in theory of computation by Venkatesan Guruswami and Mahdi Cheraghchi
 * Information Theory by Iain Murray
 * Information Theory by Cong Ling
 * Network Information Theory by Abbas El Gamal

Books
See also Textbooks.

Introductory

 * Moser, S. M., & Chen, P. N. (2012). A Student's Guide to Coding and Information Theory. Cambridge University Press.
 * Gray, R. M. (2011). Entropy and Information Theory. Springer.
 * Yeung, R. W. (2008). Information Theory and Network Coding. Springer.
 * Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. John Wiley & Sons.

Specialized

 * El Gamal, A., & Kim, Y. H. (2011). Network Information Theory. Cambridge University Press.
 * Merhav, N. (2010). Lecture Notes on Information Theory and Statistical Physics. Foundations and Trends® in Communications and Information Theory 6(1-2): 1-212.
 * Anderson, D. R. (2008). "Chapter 3: Information Theory and Entropy". Model Based Inference in the Life Sciences. Springer New York.
 * MacKay, D. J. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press.

Scholarly Articles

 * Louizos, C., Ullrich, K., & Welling, M. (2017). Bayesian Compression for Deep Learning. In Advances in Neural Information Processing Systems (pp. 3290-3300).
 * Ullrich, K., Meeds, E., & Welling, M. (2017). Soft Weight-Sharing for Neural Network Compression. arXiv preprint arXiv:1702.04008.
 * Molchanov, D., Ashukha, A., & Vetrov, D. (2017). Variational Dropout Sparsifies Deep Neural Networks. arXiv preprint arXiv:1701.05369.
 * Wen, W., Wu, C., Wang, Y., Chen, Y., & Li, H. (2016). Learning Structured Sparsity in Deep Neural Networks. In Advances in Neural Information Processing Systems (pp. 2074-2082).
 * Han, S., Mao, H., & Dally, W. J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint arXiv:1510.00149.
 * Steinruecken, C. (2014). Lossless Data Compression. PhD Diss., University of Cambridge.
 * Alajaji, F., & Chen, P. N. (2013). Lecture Notes in Information Theory: Part I.
 * Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information Bottleneck Method. arXiv preprint physics/0004057.

Software

 * Information Theory Toolbox - MATLAB
 * Octave-Information_Theory - Octave
 * Module pyentropy - Python
 * List of Compression Algorithms - Python
 * Module PyNLPl.statistics - Python
 * Information Theory and Signal Processing Library (libit) - C
 * NSB Entropy Estimation

Other Resources

 * Information Theory - Google Scholar Metrics (Top Publications)
 * Information Theory - Nature
 * Video Tutorials - Youtube channel of 'Mathematical Monk'
 * Soft weight-sharing for Neural Network Compression - Github
 * Bayesian Compression for Deep Learning - Github
 * Dynamic Network Surgery - Github
 * Software by IEEE Information Theory Society
 * Programming notes for Information Theory
 * Information Theory by Wikiversity
 * Information Theory - Notebooks