Information Theory - BKJackson/BKJackson_Wiki GitHub Wiki
Articles, Tutorials
Multiscale Entropy Analysis Tutorial Costa, Goldberger, and Peng.
Sample Entropy Wikipedia
Intro to Cross-Entropy Loss Python notebook. By Rob DiPietro.
Information Gain Slides by Andrew Moore, CMU.
How to unit test randomized algorithms using information entropy and t-test M. Faridani
Relationship between softmax and cross-entropy in neural networks
Expectation Maximization
EM Algorithm
Maximum Likelihood with Complete Information
Maximum Likelihood with Incomplete Information
Videos
9th Annual Shannon Memorial Lecture - Prof. Thomas M. Cover Uploaded October 3, 2011.
Journals
Entropy - Open Access Journal
Tsallis Entropy - Entropy Journal Special Issue
Quantum Information Nature Partner Journal, Mostly Open Source
Research Groups & Institutes
Claude Shannon
Claude Shannon: Tinkerer, Prankster, and Father of Information Theory IEEE Spectrum, Apr. 27, 2016.
Courses
MIT Information and Entropy Seth Lloyd and Paul Penfield. (2008)
MIT Information Theory Muriel Medard. (2010)
MIT Great Ideas in Theoretical Computer Science Scott Aaronson. (2008) An introduction to some of the central ideas of theoretical computer science.
MIT Automata, Computability, and Complexity Scott Aaronson. (2011) An introduction to some of the central ideas of theoretical computer science beginning in antiquity. (Similar to 2008 course.)
MIT Advanced Complexity Theory Daniel Spielman. (2001)
Complexity Theory Courses Santa Fe Institute Summer School.
Readings, SFI Complex Systems Summer School Tom Carter.
Statistical Mechanics: Algorithms and Computations Werner Krauth, Ecole normale superieure. Coursera course.
Books
Cover, Thomas M., and Thomas, Joy A. (2006) Elements of Information Theory, 2nd Edition. Hoboken, New Jersey: John Wiley & Sons, Inc. Solutions - No Peeking!
Information Theory, Inference, and Learning Algorithms David MacKay