Exercise 4.3: PDF Comparison with Regard to Differential Entropy

$h(X)$ for four density functions

The adjacent table shows the comparison result with respect to the differential entropy $h(X)$ for

the uniform distribution ⇒ $f_X(x) = f_1(x)$:

$$f_1(x) = \left\{ \begin{array}{c} 1/(2A) \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\ {\rm else} \\ \end{array} ,$$

the triangular distribution ⇒ $f_X(x) = f_2(x)$:

$$f_2(x) = \left\{ \begin{array}{c} 1/A \cdot \big [1 - |x|/A \big ] \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\ {\rm else} \\ \end{array} ,$$

the Laplace distribution ⇒ $f_X(x) = f_3(x)$:

$$f_3(x) = \lambda/2 \cdot {\rm e}^{-\lambda \hspace{0.05cm} \cdot \hspace{0.05cm}|x|}\hspace{0.05cm}.$$

The values for the Gaussian distribution ⇒ $f_X(x) = f_4(x)$ with

$$f_4(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \cdot {\rm e}^{ - \hspace{0.05cm}{x ^2}/{(2 \sigma^2})}$$

are not yet entered here. These are to be determined in subtasks (1) to (3) .

All probability density functions considered here are

symmetric about $x = 0$ ⇒ $f_X(-x) = f_X(x)$
and thus zero mean ⇒ $m_1 = 0$.

In all cases considered here, the differential entropy can be represented as follows:

Under the constraint $|X| ≤ A$ ⇒ peak constraint:

$$h(X) = {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm A} \cdot A) \hspace{0.05cm},$$

Under the constraint ${\rm E}\big [|X – m_1|^2 \big ] ≤ σ^2$ ⇒ power constraint:

$$h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2) \hspace{0.05cm}.$$

The larger the respective parameter ${\it \Gamma}_{\hspace{-0.01cm}\rm A}$ or ${\it \Gamma}_{\hspace{-0.01cm}\rm L}$ is, the more favorable is the present PDF in terms of differential entropy for the agreed constraint.

Hints:

The exercise belongs to the chapter Differential Entropy.
Useful hints for solving this task can be found in particular on the pages

Differential entropy of some peak-constrained random variables sowie

Differential entropy of some power-constrained random variables.

Questions

Solution

(1) We assume the zero mean Gaussian PDF:

$$f_X(x) = f_4(x) =A \cdot {\rm exp} [ - \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}] \hspace{0.5cm}{\rm mit}\hspace{0.5cm} A = \frac{1}{\sqrt{2\pi \sigma^2}}\hspace{0.05cm}.$$

Logarithmizing this function, the result is proposed solution 1:

$${\rm ln}\hspace{0.1cm} \big [f_X(x) \big ] = {\rm ln}\hspace{0.1cm}(A) + {\rm ln}\hspace{0.1cm}\left [{\rm exp} ( - \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}) \right ] = {\rm ln}\hspace{0.1cm}(A) - \frac{x ^2}{2 \sigma^2}\hspace{0.05cm}.$$

(2) Both proposed solutions are correct.

Using the result from (1) , we obtain for the differential entropy in "nat":

$$h_{\rm nat}(X)= -\hspace{-0.1cm} \int_{-\infty}^{+\infty} \hspace{-0.15cm} f_X(x) \cdot {\rm ln} \hspace{0.1cm} [f_X(x)] \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A) \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm} f_X(x) \hspace{0.1cm}{\rm d}x + \frac{1}{2 \sigma^2} \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm} x^2 \cdot f_X(x) \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A) + {1}/{2} \hspace{0.05cm}.$$

Here it is taken into account that the first integral is equal to $1$ (PDF area).
The second integral also gives the variance $\sigma^2$ an (if, as here, the equal part $m_1 = 0$ ).
Substituting the abbreviation variable $A$, we obtain:

$$h_{\rm nat}(X) \hspace{-0.15cm} = \hspace{-0.15cm} - {\rm ln}\hspace{0.05cm}\left (\frac{1}{\sqrt{2\pi \sigma^2}} \right ) + {1}/{2} = {1}/{2}\cdot {\rm ln}\hspace{0.05cm}\left ({2\pi \sigma^2} \right ) + {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ( {\rm e} \right ) = {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right ) \hspace{0.05cm}.$$

If the differential entropy $h(X)$ is not to be given in "nat" but in "bit", choose base 2 for the logarithm: $2$ :

$$h_{\rm bit}(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right ) \hspace{0.05cm}.$$

(3) Thus, according to the implicit definition $h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2)$ , the parameter is:

$${\it \Gamma}_{\rm L} = 2\pi {\rm e} \hspace{0.15cm}\underline{\approx 17.08} \hspace{0.05cm}.$$

(4) We now consider a Gaussian probability density function with mean $m_1$:

$$f_X(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \cdot {\rm exp}\left [ - \hspace{0.05cm}\frac{(x -m_1)^2}{2 \sigma^2} \right ] \hspace{0.05cm}.$$

The second moment $m_2 = {\rm E}\big [X ^2 \big ]$ can also be called the power $P$ , while for the variance holds (is also the second central moment):

$$\sigma^2 = {\rm E}\big [|X – m_1|^2 \big ] = \mu_2.$$

According to Steiner's theorem, $P = m_2 = m_1^2 + \sigma^2$. Thus, assuming $m_1 = \sigma = 1$ ist somit $\underline{P/\sigma^2 = 2}$.

Due to the DC component, the power is indeed doubled. However, this does not change anything in the differential entropy. Thus, it is still valid:

$$h(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )= {1}/{2} \cdot {\rm log}_2\hspace{0.05cm} (17.08)\hspace{0.15cm}\underline{\approx 2.047\,{\rm bit}} \hspace{0.05cm}.$$

Completed results table for $h(X)$

(5) The numerical values of the characteristics ${\it \Gamma}_{\rm L}$ and ${\it \Gamma}_{\rm A}$ are also entered in the completed table.

A probability density function $f_X(x)$ is always particularly favorable under power constraints if the value ${\it \Gamma}_{\rm L}$ (right column) is as large as possible. Then the differential entropy $h(X)$ is also large.

The numerical results can be interpreted as follows:

As is proved in thetheory part, the Gaussian distribution $f_4(x)$ leads here to the largest possible ${\it \Gamma}_{\rm L} ≈ 17.08$ ⇒ the proposed solution 1 is correct (the value in the last column is marked in red).
For the uniform distribution $f_1(x)$ the parameter ${\it \Gamma}_{\rm L} = 12$ is the smallest in the whole table ⇒ the proposed solution 2 is wrong.
The triangular distribution $f_2(x)$ is with ${\it \Gamma}_{\rm L} = 16.31$ günstiger als die Gleichverteilung ⇒ the proposed solution 3 is wrong.
The triangular distribution $f_2(x)$ is also better than the Laplace distribution $f_2(x) \ \ ({\it \Gamma}_{\rm L} = 14.78)$ ⇒ the proposed solution 4 is correct.

(6) A PDF $f_X(x)$ is favorable in terms of differential entropy $h(X)$ under the peak constraint ⇒ $|X| ≤ A$, if the weighting factor ${\it \Gamma}_{\rm A}$ (middle column) is as large as possible:

As shown in the theory section, the uniform distribution $f_1(x)$ here leads to the largest possible ${\it \Gamma}_{\rm A}= 2$ ⇒ the proposed solution 2 is correct (the value in the middle column is marked in red).
The triangular distribution $f_2(x)$ , which is also peak-constrained, is characterized by a somewhat smaller ${\it \Gamma}_{\rm A}= 1.649$ ⇒ dthe proposed solution 3 is incorrect.
The Gaussian distribution $f_4(x)$ is infinitely extended. A peak constraint on $|X| ≤ A$ here leads to Dirac functions in the PDF ⇒ $h(X) \to - \infty$, see sample solution to Exercise 4.2Z, subtask (4).
The same would be true for the Laplace distribution $f_3(x)$ .

	The Gaussian distribution ⇒ $f_4(x)$ leads to the maximum $h(X)$.
	The uniform distribution ⇒ $f_1(x)$ leads to the maximum $h(X)$.
	The triangular PDF ⇒ $f_2(x)$ is very unfavorable because it is peak-constrained.
	The triangular PDF ⇒ $f_2(x)$ is more favorable than the Laplace distribution ⇒ $f_3(x)$.

	Es gilt: $\ln \big[f_X(x) \big] = \ln (A) - x^2/(2 \sigma^2)$ mit $A = f_X(x=0)$.
	Es gilt: $\ln \big [f_X(x) \big] = A - \ln (x^2/(2 \sigma^2)$ mit $A = f_X(x=0)$.

	It holds: $h(X)= 1/2 \cdot \ln (2\pi\hspace{0.05cm}{\rm e}\hspace{0.05cm}\sigma^2)$ with the pseudo-unit "nat".
	It holds: $h(X)= 1/2 \cdot \log_2 (2\pi\hspace{0.05cm}{\rm e}\hspace{0.05cm}\sigma^2)$ with the pseudo-unit "bit".

	a Gaussian WDF ⇒ $f_4(x)$ followed by a constraint ⇒ $\|X\| ≤ A$,
	the uniform distribution ⇒ $f_1(x)$,
	the triangular distribution ⇒ $f_2(x)$.