Information Theory - shivamvats/notes GitHub Wiki

Let p be the probability associated with the occurrence of an event. If p is low, but we observe the event, this occurrence provides high surprise/information to us. Hence, we can mathematically define information as being inversely related to the known probability of its occurrence.

For various mathematical reasons, the definition used is:

information(E) = log2(1/p(E)) bits

Entropy of a set of events then is just the expected number of bits required to code an event in the set or it is the average information provided by every event in the set.