Difference between revisions of "Aufgaben:Exercise 4.2: Triangular PDF"

From LNTwww
(Die Seite wurde neu angelegt: „ {{quiz-Header|Buchseite=*Buch*/*Kapitel* }} [[File:|right|]] ===Fragebogen=== <quiz display=simple> {Multiple-Choice Frage |type="[]"} - Falsch + Richtig…“)
 
m (Text replacement - "value-continuous" to "continuous")
 
(28 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
{{quiz-Header|Buchseite=Information_Theory/Differential_Entropy
 +
}}
 +
 +
[[File:P_ID2865__Inf_A_4_2.png|right|frame|Two triangular PDFs]]
 +
Two probability density functions&nbsp; $\rm (PDF)$&nbsp; with triangular shapes are considered.
 +
* The random variable&nbsp;  $X$&nbsp;  is limited to the range from&nbsp;  $0$&nbsp;  to&nbsp;  $1$&nbsp;,&nbsp;  and it holds for the PDF (upper sketch):
 +
:$$f_X(x) = \left\{ \begin{array}{c} 2x \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} 0 \le x \le 1 \\    {\rm else} \\ \end{array}
 +
\hspace{0.05cm}.$$
 +
* According to the lower sketch, the random variable&nbsp;  $Y$&nbsp;  has the following PDF:
 +
:$$f_Y(y) = \left\{ \begin{array}{c} 1 - |\hspace{0.03cm}y\hspace{0.03cm}| \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} |\hspace{0.03cm}y\hspace{0.03cm}| \le 1 \\    {\rm else} \\ \end{array}
 +
\hspace{0.05cm}.$$
 +
 +
For both random variables, the&nbsp;  [[Information_Theory/Differentielle_Entropie|differential  entropy]]&nbsp; is to be determined in each case.
 +
 +
For example, the corresponding equation for the random variable&nbsp;  $X$&nbsp; is:
 +
:$$h(X) =
 +
\hspace{0.1cm} - \hspace{-0.45cm} \int\limits_{{\rm supp}\hspace{0.03cm}(\hspace{-0.03cm}f_X)} \hspace{-0.35cm}  f_X(x) \cdot {\rm log} \hspace{0.1cm} \big [ f_X(x) \big ] \hspace{0.1cm}{\rm d}x
 +
\hspace{0.6cm}{\rm with}\hspace{0.6cm} {\rm supp}(f_X) = \{ x\text{:} \  f_X(x) > 0 \}
 +
\hspace{0.05cm}.$$
 +
*If the&nbsp;  "natural logarithm",&nbsp; the pseudo-unit&nbsp;  "nat"&nbsp;  must be added.
 +
*If, on the other hand, the result is asked in&nbsp;  "bit"&nbsp; then the&nbsp;  "dual logarithm" &nbsp; &#8658; &nbsp; "$\log_2$"&nbsp; is to be used.
 +
 +
 +
In the fourth subtask, the new random variable &nbsp;$Z = A \cdot Y$&nbsp; is considered. Here,&nbsp; the PDF parameter&nbsp; $A$&nbsp; is to be determined in such a way that the differential entropy of the new random variable&nbsp; $Z$&nbsp; yields exactly&nbsp; $1$&nbsp; bit :<br>
 +
:$$h(Z) = h (A \cdot Y) =  h (Y)  + {\rm log}_2 \hspace{0.1cm} (A) = 1\ {\rm bit} \hspace{0.05cm}.$$
 +
 +
 +
 +
 +
 +
  
{{quiz-Header|Buchseite=*Buch*/*Kapitel*
 
}}
 
  
[[File:|right|]]
+
Hints:
 +
*The task belongs to the chapter&nbsp; [[Information_Theory/Differentielle_Entropie|Differential Entropy]].
 +
*Useful hints for solving this task and further information on continuous random variables can be found in the third chapter "Continuous Random Variables" of the book&nbsp;  [[Theory of Stochastic Signals]].
 +
 +
*Given the following indefinite integral:
 +
:$$\int  \xi \cdot {\rm ln} \hspace{0.1cm} (\xi)\hspace{0.1cm}{\rm d}\xi =
 +
\xi^2 \cdot \big [1/2 \cdot {{\rm ln} \hspace{0.1cm} (\xi)} -
 +
{1}/{4}\big ] \hspace{0.05cm}.$$
  
  
===Fragebogen===
 
  
 +
===Questions===
 
<quiz display=simple>
 
<quiz display=simple>
{Multiple-Choice Frage
+
{Calculate the differential entropy of the random variable&nbsp; $X$&nbsp; in&nbsp; "nat".
|type="[]"}
+
|type="{}"}
- Falsch
+
$h(X) \ = \ $ { -0.199--0.187 } $\ \rm nat$
+ Richtig
+
 
  
 +
{What result is obtained with the pseudo-unit&nbsp; "bit"?
 +
|type="{}"}
 +
$h(X) \ = \ $ { -0.288--0.270 } $\ \rm bit$
  
{Input-Box Frage
+
{Calculate the differential entropy of the random variable&nbsp; $Y$.
 
|type="{}"}
 
|type="{}"}
$\alpha$ = { 0.3 }
+
$h(Y) \ = \ $ { 0.721 3% } $\ \rm bit$
  
 +
{Determine the PDF parameter&nbsp; $A$&nbsp; such that&nbsp; $\underline{h(Z) = h (A \cdot Y) = 1 \ \rm bit}$&nbsp;.
 +
|type="{}"}
 +
$A\ = $ { 1.213 3% }
  
  
 
</quiz>
 
</quiz>
  
===Musterlösung===
+
===Solution===
 
{{ML-Kopf}}
 
{{ML-Kopf}}
'''1.'''
+
'''(1)'''&nbsp; For the probability density function, in the range&nbsp; $0 \le X \le 1$&nbsp;, it is agreed that:
'''2.'''
+
:$$f_X(x) = 2x = C \cdot x
'''3.'''
+
\hspace{0.05cm}.$$
'''4.'''
+
*Here we have replaced&nbsp; "2"&nbsp; by&nbsp; $C$&nbsp; &nbsp; &#8658; &nbsp; generalization in order to be able to use the following calculation again in subtask&nbsp; $(3)$&nbsp;.
'''5.'''
+
 
'''6.'''
+
*Since the differential entropy is sought in&nbsp; "nat",&nbsp; we use the natural logarithm.&nbsp; With the substitution&nbsp; $\xi = C \cdot x$&nbsp; we obtain:
'''7.'''
+
:$$h_{\rm nat}(X) = \hspace{0.1cm} - \int_{0}^{1} \hspace{0.1cm}  C \cdot x \cdot {\rm ln} \hspace{0.1cm} \big[ C \cdot x \big] \hspace{0.1cm}{\rm d}x =
 +
\hspace{0.1cm} - \hspace{0.1cm}\frac{1}{C} \cdot \int_{0}^{C} \hspace{0.1cm}  \xi \cdot {\rm ln} \hspace{0.1cm} [ \xi ] \hspace{0.1cm}{\rm d}\xi
 +
  =  - \hspace{0.1cm}\frac{\xi^2}{C} \cdot 
 +
\left [ \frac{{\rm ln} \hspace{0.1cm} (\xi)}{2} -
 +
\frac{1}{4}\right ]_{\xi = 0}^{\xi = C}
 +
\hspace{0.05cm}$$
 +
*Here the indefinite integral given in the front was used.&nbsp; After inserting the limits, considering&nbsp; $C=2$,&nbsp; we obtain::
 +
:$$h_{\rm nat}(X) =
 +
- C/2 \cdot 
 +
\big [ {\rm ln} \hspace{0.1cm} (C) - 1/2
 +
\big ]
 +
= - {\rm ln} \hspace{0.1cm} (2) + 1/2 =
 +
- {\rm ln} \hspace{0.1cm} (2)
 +
+ 1/2 \cdot {\rm ln} \hspace{0.1cm} ({\rm e})
 +
  =  {\rm ln} \hspace{0.1cm} (\sqrt{\rm e}/2)\hspace{0.05cm} = - 0.193
 +
\hspace{0.3cm} \Rightarrow\hspace{0.3cm}
 +
h(X) 
 +
\hspace{0.15cm}\underline {= - 0.193\,{\rm nat}}
 +
\hspace{0.05cm}.$$
 +
 
 +
 
 +
 
 +
'''(2)'''&nbsp; In general:
 +
[[File:P_ID2866__Inf_A_4_2c.png|right|frame|To calculate&nbsp; $h(Y)$]]
 +
 
 +
:$$h_{\rm bit}(X) = \frac{h_{\rm nat}(X)}{{\rm ln} \hspace{0.1cm} (2)\,{\rm nat/bit}} = - 0.279
 +
\hspace{0.3cm} \Rightarrow\hspace{0.3cm}
 +
h(X)
 +
\hspace{0.15cm}\underline {= - 0.279\,{\rm bit}}
 +
\hspace{0.05cm}.$$
 +
*You can save this conversion if you directly replace&nbsp; $(1)$&nbsp; direct&nbsp; "ln"&nbsp; by&nbsp; "log<sub>2</sub>"&nbsp; already in the analytical result of subtask:
 +
 
 +
:$$h(X) = \  {\rm log}_2 \hspace{0.1cm} (\sqrt{\rm e}/2)\hspace{0.05cm}, \hspace{1.3cm}
 +
{\rm pseudo-unit\hspace{-0.1cm}:\hspace{0.15cm} bit}
 +
\hspace{0.05cm}.$$
 +
 
 +
 
 +
 
 +
'''(3)'''&nbsp; We again use the natural logarithm and divide the integral into two partial integrals:
 +
:$$h(Y) =
 +
\hspace{0.1cm} - \hspace{-0.45cm} \int\limits_{{\rm supp}
 +
\hspace{0.03cm}( \hspace{-0.03cm}f_Y)} \hspace{-0.35cm}  f_Y(y) \cdot {\rm ln} \hspace{0.1cm} \big[ f_Y(y) \big] \hspace{0.1cm}{\rm d}y = I_{\rm neg} + I_{\rm pos}
 +
\hspace{0.05cm}.$$
 +
 
 +
*The first integral for the range&nbsp; $-1 \le y \le 0$&nbsp; is identical in form to that of subtask&nbsp; $(1)$&nbsp; and only shifted with respect to it, which does not affect the result.
 +
*Now the height&nbsp; $C = 1$&nbsp; instead of&nbsp; $C = 2$&nbsp; has to be considered:
 +
:$$I_{\rm neg} =- C/2 \cdot 
 +
\big [ {\rm ln} \hspace{0.1cm} (C) - 1/2
 +
\big ] = -1/2 \cdot 
 +
\big [ {\rm ln} \hspace{0.1cm} (1) - 1/2 \cdot {\rm ln} \hspace{0.1cm} ({\rm e}) \big ]= 1/4 \cdot
 +
{\rm ln} \hspace{0.1cm} ({\rm e}) 
 +
\hspace{0.05cm}.$$
 +
 
 +
*The second integrand is identical to the first except for a shift and reflection.&nbsp; Moreover, the integration intervals do not overlap &nbsp; &#8658; &nbsp; $I_{\rm pos} = I_{\rm neg}$:
 +
:$$h_{\rm nat}(Y)  = 2 \cdot I_{\rm neg} =  1/2 \cdot
 +
{\rm ln} \hspace{0.1cm} ({\rm e}) = {\rm ln} \hspace{0.1cm} (\sqrt{\rm e}) \hspace{0.3cm}
 +
\Rightarrow\hspace{0.3cm}h_{\rm bit}(Y)  = {\rm log}_2 \hspace{0.1cm} (\sqrt{\rm e})
 +
\hspace{0.3cm} \Rightarrow\hspace{0.3cm}
 +
h(Y)  = {\rm log}_2 \hspace{0.1cm} (1.649)\hspace{0.15cm}\underline {= 0.721\,{\rm bit}}\hspace{0.05cm}.$$
 +
 
 +
 
 +
 
 +
'''(4)'''&nbsp; For the differential entropy of the random variable&nbsp; $Z = A \cdot Y$&nbsp; holds in general:
 +
:$$h(Z)  = h(A \cdot Y)  = h(Y) + {\rm log}_2 \hspace{0.1cm} (A) \hspace{0.05cm}.$$
 +
*Thus, from the requiremen&nbsp; $h(Z) = 1 \ \rm bit$&nbsp; and the result of subtask&nbsp; $(3)$&nbsp; follows:
 +
:$${\rm log}_2 \hspace{0.1cm} (A) = 1\,{\rm bit} - 0.721 \,{\rm bit} = 0.279 \,{\rm bit}
 +
\hspace{0.3cm} \Rightarrow\hspace{0.3cm} A = 2^{0.279}\hspace{0.15cm}\underline
 +
{= 1.213}
 +
\hspace{0.05cm}.$$
 +
 
 
{{ML-Fuß}}
 
{{ML-Fuß}}
  
  
 
+
[[Category:Information Theory: Exercises|^4.1  Differential Entropy^]]
[[Category:Aufgaben zu Lineare zeitinvariante Systeme|^Kapitelx^]]
 

Latest revision as of 09:27, 11 October 2021

Two triangular PDFs

Two probability density functions  $\rm (PDF)$  with triangular shapes are considered.

  • The random variable  $X$  is limited to the range from  $0$  to  $1$ ,  and it holds for the PDF (upper sketch):
$$f_X(x) = \left\{ \begin{array}{c} 2x \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} 0 \le x \le 1 \\ {\rm else} \\ \end{array} \hspace{0.05cm}.$$
  • According to the lower sketch, the random variable  $Y$  has the following PDF:
$$f_Y(y) = \left\{ \begin{array}{c} 1 - |\hspace{0.03cm}y\hspace{0.03cm}| \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} |\hspace{0.03cm}y\hspace{0.03cm}| \le 1 \\ {\rm else} \\ \end{array} \hspace{0.05cm}.$$

For both random variables, the  differential entropy  is to be determined in each case.

For example, the corresponding equation for the random variable  $X$  is:

$$h(X) = \hspace{0.1cm} - \hspace{-0.45cm} \int\limits_{{\rm supp}\hspace{0.03cm}(\hspace{-0.03cm}f_X)} \hspace{-0.35cm} f_X(x) \cdot {\rm log} \hspace{0.1cm} \big [ f_X(x) \big ] \hspace{0.1cm}{\rm d}x \hspace{0.6cm}{\rm with}\hspace{0.6cm} {\rm supp}(f_X) = \{ x\text{:} \ f_X(x) > 0 \} \hspace{0.05cm}.$$
  • If the  "natural logarithm",  the pseudo-unit  "nat"  must be added.
  • If, on the other hand, the result is asked in  "bit"  then the  "dual logarithm"   ⇒   "$\log_2$"  is to be used.


In the fourth subtask, the new random variable  $Z = A \cdot Y$  is considered. Here,  the PDF parameter  $A$  is to be determined in such a way that the differential entropy of the new random variable  $Z$  yields exactly  $1$  bit :

$$h(Z) = h (A \cdot Y) = h (Y) + {\rm log}_2 \hspace{0.1cm} (A) = 1\ {\rm bit} \hspace{0.05cm}.$$





Hints:

  • The task belongs to the chapter  Differential Entropy.
  • Useful hints for solving this task and further information on continuous random variables can be found in the third chapter "Continuous Random Variables" of the book  Theory of Stochastic Signals.
  • Given the following indefinite integral:
$$\int \xi \cdot {\rm ln} \hspace{0.1cm} (\xi)\hspace{0.1cm}{\rm d}\xi = \xi^2 \cdot \big [1/2 \cdot {{\rm ln} \hspace{0.1cm} (\xi)} - {1}/{4}\big ] \hspace{0.05cm}.$$


Questions

1

Calculate the differential entropy of the random variable  $X$  in  "nat".

$h(X) \ = \ $

$\ \rm nat$

2

What result is obtained with the pseudo-unit  "bit"?

$h(X) \ = \ $

$\ \rm bit$

3

Calculate the differential entropy of the random variable  $Y$.

$h(Y) \ = \ $

$\ \rm bit$

4

Determine the PDF parameter  $A$  such that  $\underline{h(Z) = h (A \cdot Y) = 1 \ \rm bit}$ .

$A\ = $


Solution

(1)  For the probability density function, in the range  $0 \le X \le 1$ , it is agreed that:

$$f_X(x) = 2x = C \cdot x \hspace{0.05cm}.$$
  • Here we have replaced  "2"  by  $C$    ⇒   generalization in order to be able to use the following calculation again in subtask  $(3)$ .
  • Since the differential entropy is sought in  "nat",  we use the natural logarithm.  With the substitution  $\xi = C \cdot x$  we obtain:
$$h_{\rm nat}(X) = \hspace{0.1cm} - \int_{0}^{1} \hspace{0.1cm} C \cdot x \cdot {\rm ln} \hspace{0.1cm} \big[ C \cdot x \big] \hspace{0.1cm}{\rm d}x = \hspace{0.1cm} - \hspace{0.1cm}\frac{1}{C} \cdot \int_{0}^{C} \hspace{0.1cm} \xi \cdot {\rm ln} \hspace{0.1cm} [ \xi ] \hspace{0.1cm}{\rm d}\xi = - \hspace{0.1cm}\frac{\xi^2}{C} \cdot \left [ \frac{{\rm ln} \hspace{0.1cm} (\xi)}{2} - \frac{1}{4}\right ]_{\xi = 0}^{\xi = C} \hspace{0.05cm}$$
  • Here the indefinite integral given in the front was used.  After inserting the limits, considering  $C=2$,  we obtain::
$$h_{\rm nat}(X) = - C/2 \cdot \big [ {\rm ln} \hspace{0.1cm} (C) - 1/2 \big ] = - {\rm ln} \hspace{0.1cm} (2) + 1/2 = - {\rm ln} \hspace{0.1cm} (2) + 1/2 \cdot {\rm ln} \hspace{0.1cm} ({\rm e}) = {\rm ln} \hspace{0.1cm} (\sqrt{\rm e}/2)\hspace{0.05cm} = - 0.193 \hspace{0.3cm} \Rightarrow\hspace{0.3cm} h(X) \hspace{0.15cm}\underline {= - 0.193\,{\rm nat}} \hspace{0.05cm}.$$


(2)  In general:

To calculate  $h(Y)$
$$h_{\rm bit}(X) = \frac{h_{\rm nat}(X)}{{\rm ln} \hspace{0.1cm} (2)\,{\rm nat/bit}} = - 0.279 \hspace{0.3cm} \Rightarrow\hspace{0.3cm} h(X) \hspace{0.15cm}\underline {= - 0.279\,{\rm bit}} \hspace{0.05cm}.$$
  • You can save this conversion if you directly replace  $(1)$  direct  "ln"  by  "log2"  already in the analytical result of subtask:
$$h(X) = \ {\rm log}_2 \hspace{0.1cm} (\sqrt{\rm e}/2)\hspace{0.05cm}, \hspace{1.3cm} {\rm pseudo-unit\hspace{-0.1cm}:\hspace{0.15cm} bit} \hspace{0.05cm}.$$


(3)  We again use the natural logarithm and divide the integral into two partial integrals:

$$h(Y) = \hspace{0.1cm} - \hspace{-0.45cm} \int\limits_{{\rm supp} \hspace{0.03cm}( \hspace{-0.03cm}f_Y)} \hspace{-0.35cm} f_Y(y) \cdot {\rm ln} \hspace{0.1cm} \big[ f_Y(y) \big] \hspace{0.1cm}{\rm d}y = I_{\rm neg} + I_{\rm pos} \hspace{0.05cm}.$$
  • The first integral for the range  $-1 \le y \le 0$  is identical in form to that of subtask  $(1)$  and only shifted with respect to it, which does not affect the result.
  • Now the height  $C = 1$  instead of  $C = 2$  has to be considered:
$$I_{\rm neg} =- C/2 \cdot \big [ {\rm ln} \hspace{0.1cm} (C) - 1/2 \big ] = -1/2 \cdot \big [ {\rm ln} \hspace{0.1cm} (1) - 1/2 \cdot {\rm ln} \hspace{0.1cm} ({\rm e}) \big ]= 1/4 \cdot {\rm ln} \hspace{0.1cm} ({\rm e}) \hspace{0.05cm}.$$
  • The second integrand is identical to the first except for a shift and reflection.  Moreover, the integration intervals do not overlap   ⇒   $I_{\rm pos} = I_{\rm neg}$:
$$h_{\rm nat}(Y) = 2 \cdot I_{\rm neg} = 1/2 \cdot {\rm ln} \hspace{0.1cm} ({\rm e}) = {\rm ln} \hspace{0.1cm} (\sqrt{\rm e}) \hspace{0.3cm} \Rightarrow\hspace{0.3cm}h_{\rm bit}(Y) = {\rm log}_2 \hspace{0.1cm} (\sqrt{\rm e}) \hspace{0.3cm} \Rightarrow\hspace{0.3cm} h(Y) = {\rm log}_2 \hspace{0.1cm} (1.649)\hspace{0.15cm}\underline {= 0.721\,{\rm bit}}\hspace{0.05cm}.$$


(4)  For the differential entropy of the random variable  $Z = A \cdot Y$  holds in general:

$$h(Z) = h(A \cdot Y) = h(Y) + {\rm log}_2 \hspace{0.1cm} (A) \hspace{0.05cm}.$$
  • Thus, from the requiremen  $h(Z) = 1 \ \rm bit$  and the result of subtask  $(3)$  follows:
$${\rm log}_2 \hspace{0.1cm} (A) = 1\,{\rm bit} - 0.721 \,{\rm bit} = 0.279 \,{\rm bit} \hspace{0.3cm} \Rightarrow\hspace{0.3cm} A = 2^{0.279}\hspace{0.15cm}\underline {= 1.213} \hspace{0.05cm}.$$