entropy example 3 - davidar/scholarpedia GitHub Wiki

Consider the unit square representing the space <math>\Omega\ ,</math> where the probability is the Lebesgue measure (i.e., the surface area), and the partition <math>\mathcal A</math> into four sets <math>A_i</math> of probabilities <math>\frac18, \frac14, \frac18, \frac12\ ,</math> respectively, as shown in <figref>EntropyFigure4.jpg</figref>.

The partition \mathcal A\ .

The information function equals <math>-\log_2\left(\frac 18\right) = 3</math> on <math>A_1</math> and <math>A_3\ ,</math> <math>-\log_2\left(\frac 14\right) = 2</math> on <math>A_2</math> and <math>-\log_2\left(\frac 12\right) = 1</math> on <math>A_4\ .</math> The entropy of <math>\mathcal A</math> equals

<math>

H(\mathcal A) = \frac18\cdot3 + \frac14\cdot2 + \frac18\cdot3 + \frac12\cdot1 = \frac74. </math>

The arrangement of questions that optimizes the expected value of the number of questions asked is the following (see <figref>EntropyFigure5.jpg</figref>):

• Question 1. Are you in the left half?

The answer no, locates <math>\omega</math> in <math>A_4</math> using one bit. Otherwise the next question is:

• Question 2. Are you in the central square of the left half?

The yes answer locates <math>\omega</math> in <math>A_2</math> using two bits. If not, the last question is:

• Question 3. Are you in the top half of the whole square?

Now yes and no locate <math>\omega</math> in <math>A_1</math> or <math>A_3\ ,</math> respectively. This takes three bits.

The arrangement of binary questions.

In this example the number of questions equals exactly the information function at every point and the expected number of question equals the entropy <math>\frac 74\ .</math> There does not exist a better arrangement of questions. Of course such accuracy is possible only when the probabilities of the sets <math>A_i</math> are powers of <math>\frac12\ .</math>