Information Theory

Shannon Entropy

Shannon Entropy $H(X)$$\large H(X)$ of a distribution is the expected amount of information in an event drawn from that distribution. It gives a lower bound on the number of bits needed on average to encode symbols drawn from a distribution P.

The Shannon Entropy measures:

• the degree of uncertainty in a value
• the amount of "surprise" in seeing this observation
• how much information is represented by this distribution

Kullback-Liebler Divergence

Kullback-Liebler Divergence $KI$$\large KI$ is a method for measuring the dissimilarity between two probability distributions $P$$\large P$ and $Q$$\large Q$. It can also be seen as the Relative Entropy measure between the two distributions.

• $KL(P||Q)\ge 0$$\large KL(P||Q) \ge 0$ and $KL(P||Q)=0$$\large KL(P||Q) = 0$ iff $P=Q$$\large P = Q$
• $KL(P||Q)$$\large KL(P||Q)$ represents the amount of information lost on approximating $Q$$\large Q$ with $P$$\large P$
• In general, $KL(P||Q)\ne KL(Q||P)$$\large KL(P||Q) \ne KL(Q||P)$

Mutual Information

Mutual Information $MI$$\large MI$ between two vectors $X$$\large X$ and $Y$$\large Y$ measures the dissimilarity between joint distribution $p(X,Y)$$\large p(X, Y)$ and factored distribution $p(X)p(Y)$$\large p(X) \: p(Y)$. Mutual Information also measures the reduction in uncertainty for one variable given a known value of the other variable.

• $MI(X,Y)\ge 0$$\large MI(X, Y) \ge 0$
• $MI(X,Y)=0$$\large MI(X, Y) = 0$ iff $X$$\large X$ and $Y$$\large Y$ are independent
• $MI(X,Y)=H(X)+H(X)-H(X,Y)$$\large MI(X, Y) = H(X) + H(X) - H(X, Y)$
• $MI(X,Y)=KL(p(X,Y),p(X)p(Y))$$\large MI(X, Y) = KL(p(X, Y), p(X) \: p(Y))$

Information Gain

Information Gain $IG$$\large IG$ measures the reduction in entropy or surprise by splitting a dataset according to a given value of a random variable.