Difference between revisions of "Aufgaben:Exercise 4.3: PDF Comparison with Regard to Differential Entropy"

From LNTwww
 
(29 intermediate revisions by 4 users not shown)
Line 1: Line 1:
  
{{quiz-Header|Buchseite=Informationstheorie/Differentielle Entropie
+
{{quiz-Header|Buchseite=Information_Theory/Differential_Entropy
 
}}
 
}}
  
[[File:P_ID2874__Inf_A_4_3.png|right|]]
+
[[File:EN_Inf_A_4_3_v2.png|right|frame|$h(X)$  for four probability density functions]]
Nebenstehende Tabelle zeigt das Vergleichsergebnis hinsichtlich der differentiellen Entropie <i>h</i>(<i>X</i>) für
+
The adjacent table shows the comparison result with respect to the differential entropy&nbsp; $h(X)$&nbsp; for
:* die <b>Gleichverteilung</b> &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>f<sub>X</sub></i>(<i>x</i>) = <i>f</i><sub>1</sub>(<i>x</i>):  
+
* the&nbsp; [[Theory_of_Stochastic_Signals/Gleichverteilte_Zufallsgrößen|uniform distribution]] &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $f_X(x) = f_1(x)$:  
$$f_1(x) = \left\{ \begin{array}{c} 1/(2A)  \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\    {\rm sonst} \\ \end{array}
+
:$$f_1(x) = \left\{ \begin{array}{c} 1/(2A)  \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\    {\rm else} \\ \end{array}
 
,$$
 
,$$
:* die <b>Dreieckverteilung</b> &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>f<sub>X</sub></i>(<i>x</i>) = <i>f</i><sub>2</sub>(<i>x</i>):
+
* the&nbsp; [[Aufgaben:Exercise_3.1Z:_Triangular_PDF|triangular distribution]] &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $f_X(x) = f_2(x)$:
$$f_2(x) = \left\{ \begin{array}{c} 1/A \cdot [1 - |x|/A] \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\    {\rm sonst} \\ \end{array}
+
:$$f_2(x) = \left\{ \begin{array}{c} 1/A \cdot \big [1 - |x|/A \big ] \\  0 \\  \end{array} \right. \begin{array}{*{20}c}  {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\    {\rm else} \\ \end{array}
 
,$$
 
,$$
:* die <b>Laplaceverteilung</b> &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>f<sub>X</sub></i>(<i>x</i>) = <i>f</i><sub>3</sub>(<i>x</i>):
+
* the&nbsp; [[Theory_of_Stochastic_Signals/Exponentially_Distributed_Random_Variables#Two-sided_exponential_distribution_-_Laplace_distribution|Laplace distribution]] &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $f_X(x) = f_3(x)$:
$$f_3(x) =  \lambda/2 \cdot {\rm exp}[-\lambda \cdot |x|]\hspace{0.05cm}.$$
+
:$$f_3(x) =  \lambda/2 \cdot {\rm e}^{-\lambda \hspace{0.05cm} \cdot \hspace{0.05cm}|x|}\hspace{0.05cm}.$$
  
Die Werte für die <b>Gaußverteilung</b> &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>f<sub>X</sub></i>(<i>x</i>) = <i>f</i><sub>4</sub>(<i>x</i>)
+
The values for the&nbsp; [[Theory_of_Stochastic_Signals/Gaußverteilte_Zufallsgrößen|Gaussian distribution]] &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $f_X(x) = f_4(x)$&nbsp; with
$$f_4(x) = \frac{1}{\sqrt{2\pi  \sigma^2}} \cdot {\rm exp} [
+
:$$f_4(x) = \frac{1}{\sqrt{2\pi  \sigma^2}} \cdot {\rm e}^{
- \hspace{0.05cm}{x ^2}/{(2 \sigma^2})]$$
+
- \hspace{0.05cm}{x ^2}/{(2 \sigma^2})}$$
sind hier noch nicht eingetragen. Diese sollen in den Teilaufgaben (a) bis (c) ermittelt werden.
+
are not yet entered here.&nbsp; These are to be determined in subtasks&nbsp; '''(1)'''&nbsp; to&nbsp; '''(3)'''&nbsp;.
  
Alle hier betrachteten Wahrscheinlichkeitsdichtefunktionen sind
+
Each probability density function&nbsp; $\rm (PDF)$&nbsp; considered here is
:* symmetrisch um <i>x</i> = 0 &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>f<sub>X</sub></i>(<i>&ndash;x</i>) = <i>f<sub>X</sub></i>(<i>x</i>)
+
* symmetric about&nbsp; $x = 0$&nbsp; &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $f_X(-x) = f_X(x)$
:* und damit mittelwertfrei &nbsp;&nbsp;&#8658;&nbsp;&nbsp; <i>m</i><sub>1</sub> = 0.
+
* and thus zero mean &nbsp;&nbsp;&#8658;&nbsp;&nbsp;$m_1 = 0$.
In allen hier betrachteten Fällen kann die differentielle Entropie wie folgt dargestellt werden:
+
 
<ul class="liste_ohne"><li>Unter der Nebenbedingung |<i>X</i>| &#8804; <i>A</i> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8658;&nbsp;&nbsp;  [http://en.lntwww.de/Informationstheorie/Differentielle_Entropie#Differentielle_Entropie_einiger_spitzenwertbegrenzter_Zufallsgr.C3.B6.C3.9Fen  '''Spitzenwertbegrenzung:'''  ]
+
 
$$h(X) = {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.1cm}\rm A} \cdot A)  
+
In all cases considered here, the differential entropy can be represented as follows:
 +
*Under the constraint&nbsp; $|X| ≤ A$ &nbsp; &#8658; &nbsp;&nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_peak_constraint|peak constraint]]&nbsp; $($German:&nbsp; "Spitzenwertbegrenzung"&nbsp; or&nbsp; "Amplitudenbegrenzung" &nbsp; &rArr; &nbsp; Identifier:&nbsp;  $\rm A)$:
 +
:$$h(X) = {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm A} \cdot A)  
 
\hspace{0.05cm},$$
 
\hspace{0.05cm},$$
<ul class="liste_ohne"><li>Unter der Nebenbedingung E[|<i>X</i>|<sup>2</sup>] &#8804; <i>&sigma;</i><sup>2</sup> &nbsp;&#8658;&nbsp; [http://en.lntwww.de/Informationstheorie/Differentielle_Entropie#Differentielle_Entropie_einiger_leistungsbegrenzter_Zufallsgr.C3.B6.C3.9Fen '''Leistungsbegrenzung:''' ]
+
*Under the constraint&nbsp; ${\rm E}\big [|X – m_1|^2 \big ] ≤ σ^2$ &nbsp; &#8658; &nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_power_constraint|power constraint]]&nbsp; $($German:&nbsp; "Leistungsbegrenzung" &nbsp; &rArr;  &nbsp; Identifier:&nbsp;  $\rm L)$:
$$h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.1cm}\rm L} \cdot \sigma^2)  
+
:$$h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2)  
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
Je größer die jeweilige Kenngröße <i>&Gamma;</i><sub>A</sub> bzw. <i>&Gamma;</i><sub>L</sub> ist, desto günstiger ist bei der vereinbarten Nebenbedingung die vorliegende WDF hinsichtlich der differentiellen Entropie.
+
The larger the respective parameter&nbsp; ${\it \Gamma}_{\hspace{-0.01cm}\rm A}$&nbsp; or&nbsp; ${\it \Gamma}_{\hspace{-0.01cm}\rm L}$&nbsp; is, the more favorable is the present PDF in terms of differential entropy for the agreed constraint.
$$$$<b>Hinweis:</b> Die Aufgabe gehört zum Themengebiet von [http://en.lntwww.de/Informationstheorie/Differentielle_Entropie '''Kapitel 4.1'''].
+
 
  
  
===Fragebogen===
+
 
 +
 
 +
 
 +
 
 +
 
 +
Hints:
 +
*The exercise belongs to the chapter&nbsp; [[Information_Theory/Differentielle_Entropie|Differential Entropy]].
 +
*Useful hints for solving this task can be found in particular on the pages
 +
::[[Information_Theory/Differentielle_Entropie#Differential_entropy_of_some_peak-constrained_random_variables|Differential entropy of some peak-constrained random variables]],&nbsp;
 +
::[[Information_Theory/Differentielle_Entropie#Differential_entropy_of_some_power-constrained_random_variables|Differential entropy of some power-constrained random variables]].
 +
 +
 
 +
 
 +
 
 +
===Questions===
  
 
<quiz display=simple>
 
<quiz display=simple>
{Welche Gleichung gilt für den Logarithmus der Gauß&ndash;WDF?
+
{Which equation is valid for the logarithm of the Gaussian PDF?
 
|type="[]"}
 
|type="[]"}
+ ln [<i>f<sub>X</sub></i>(<i>x</i>)] = ln <i>A</i> &ndash; <i>x</i><sup>2</sup>/(2 <i>&sigma;</i><sup>2</sup>) &nbsp;&nbsp;&nbsp; mit &nbsp;&nbsp;&nbsp; <i>A</i> = <i>f<sub>X</sub></i>(<i>x</i> = 0),
+
+ It holds: &nbsp; $\ln \big[f_X(x) \big] = \ln (A) - x^2/(2 \sigma^2)$ &nbsp; with &nbsp; $A = f_X(x=0)$.
- ln [<i>f<sub>X</sub></i>(<i>x</i>)] = <i>A</i> &ndash; ln (<i>x</i><sup>2</sup>/<i>&sigma;</i><sup>2</sup>) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mit &nbsp;&nbsp;&nbsp; <i>A</i> = <i>f<sub>X</sub></i>(<i>x</i> = 0).
+
- Es It holds: &nbsp; $\ln \big [f_X(x) \big] = A - \ln (x^2/(2 \sigma^2)$ &nbsp; with &nbsp; $A = f_X(x=0)$.
  
{Welche Gleichung gilt für die differentielle Entropie  der Gauß&ndash;WDF?
+
{Which equation holds for the differential entropy of the Gaussian PDF?
 
|type="[]"}
 
|type="[]"}
+ <i>h</i>(<i>X</i>) = 1/2 &middot; ln (2<i>&pi;</i>e<i>&sigma;</i><sup>2</sup>) mit Pseudoeinheit &bdquo;nat&rdquo;.
+
+ It holds: &nbsp; $h(X)= 1/2 \cdot \ln (2\pi\hspace{0.05cm}{\rm e}\hspace{0.01cm}\cdot\hspace{0.01cm}\sigma^2)$&nbsp; with the pseudo-unit&nbsp; "nat".
+ <i>h</i>(<i>X</i>) = 1/2 &middot; log<sub>2</sub> (2<i>&pi;</i>e<i>&sigma;</i><sup>2</sup>) mit Pseudoeinheit &bdquo;bit&rdquo;.
+
+ It holds: &nbsp; $h(X)= 1/2 \cdot \log_2 (2\pi\hspace{0.05cm}{\rm e}\hspace{0.01cm}\cdot\hspace{0.01cm}\sigma^2)$&nbsp; with the pseudo-unit&nbsp; "bit".
  
{Ergänzen Sie den fehlenden Eintrag (Gauß) in obiger Tabelle.
+
{Complete the missing entry for the Gaussian PDF in the above table.
 
|type="{}"}
 
|type="{}"}
$ ΓL$ = { 17.08 3% }
+
${\it \Gamma}_{\rm L} \ = \ $ { 17.08 3% }
  
{Welche Werte erhält man für die Gauß&ndash;WDF mit Gleichanteil <i>m</i><sub>1</sub> = <i>&sigma;</i> = 1?
+
{What values are obtained for the Gaussian PDF with the DC component &nbsp;$m_1 = \sigma = 1$?
 
|type="{}"}
 
|type="{}"}
$ P/σ2$ = { 2 3% }
+
$P/\sigma^2 \ = \ $ { 2 3% }
$ h(X)$ = { 2.047 3% }
+
$h(X) \ = \ $ { 2.047 3% } $\ \rm bit$
  
{Welche der Aussagen stimmen für die differentielle Entropie <i>h</i>(<i>X</i>) unter der Nebenbedingung &bdquo;Leistungsbegrenzung&rdquo;?
+
{Which of the statements are true for the differential entropy&nbsp; $h(X)$&nbsp; considering the&nbsp; "power constraint"&nbsp; ${\rm E}\big[|X – m_1|^2\big] ≤ σ^2$?
 
|type="[]"}
 
|type="[]"}
+ Die Gaußverteilung führt zum maximalen <i>h</i>(<i>X</i>).
+
+ The Gaussian PDF &nbsp; &rArr; &nbsp; $f_4(x)$&nbsp; leads to the maximum&nbsp; $h(X)$.
- Die Gleichverteilung führt zum maximalen <i>h</i>(<i>X</i>).
+
- The uniform PDF &nbsp; &rArr; &nbsp; $f_1(x)$&nbsp; leads to the maximum&nbsp; $h(X)$.
- Die Dreieck&ndash;WDF ist sehr ungünstig, da spitzenwertbegrenzt.
+
- The triangular PDF &nbsp; &rArr; &nbsp; $f_2(x)$&nbsp; is very unfavorable because it is peak-constrained.
 +
+ The triangular PDF &nbsp; &rArr; &nbsp; $f_2(x)$&nbsp; is more favorable than the Laplace PDF &nbsp; &rArr; &nbsp;  $f_3(x)$.  
  
{Welche der Aussagen stimmen bei &bdquo;Spitzenwertbegrenzung&rdquo; auf den Bereich <nobr>|<i>X</i>| &#8804; <i>A</i>?</nobr> Die maximale differentielle Entropie <i>h</i>(<i>X</i>) ergibt sich für
+
{Which of the statements are true for&nbsp; "peak constraint"&nbsp; to the range&nbsp$|X| ≤ A$.&nbsp; The maximum differential entropy&nbsp$h(X)$&nbsp; is obtained for
 
|type="[]"}
 
|type="[]"}
- eine Gauß&ndash;WDF mit anschließender Begrenzung &nbsp;&#8658;&nbsp; |<i>X</i>| &#8804; <i>A</i>,
+
- a Gaussian PDF &nbsp; &rArr; &nbsp; $f_4(x)$&nbsp; followed by a constraint &nbsp; &#8658; &nbsp;$|X| A$,
+ die Gleichverteilung,
+
+ the uniform PDF &nbsp; &rArr; &nbsp; $f_1(x)$,
- die Dreieckverteilung.
+
- the triangular PDF &nbsp; &rArr; &nbsp; $f_2(x)$.
  
 
</quiz>
 
</quiz>
  
===Musterlösung===
+
===Solution===
 
{{ML-Kopf}}
 
{{ML-Kopf}}
<b>a)</b>&nbsp;&nbsp;Wir gehen von der mittelwertfreien Gauß&ndash;WDF aus:
+
'''(1)'''&nbsp; We assume the zero mean Gaussian PDF:
$$f_X(x) = f_4(x) =A \cdot {\rm exp} [  
+
:$$f_X(x) = f_4(x) =A \cdot {\rm exp} [  
 
- \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}]
 
- \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}]
\hspace{0.5cm}{\rm mit}\hspace{0.5cm}
+
\hspace{0.5cm}{\rm with}\hspace{0.5cm}
 
A = \frac{1}{\sqrt{2\pi  \sigma^2}}\hspace{0.05cm}.$$
 
A = \frac{1}{\sqrt{2\pi  \sigma^2}}\hspace{0.05cm}.$$
Logarithmiert man diese Funktion, so erhält man als Ergebnis den <u>Lösungsvorschlag 1</u>:
+
*Logarithmizing this function, the result is <u>proposed solution 1</u>:
$${\rm ln}\hspace{0.1cm} \left [f_X(x) \right ] = {\rm ln}\hspace{0.1cm}(A) +
+
:$${\rm ln}\hspace{0.1cm} \big [f_X(x) \big ] = {\rm ln}\hspace{0.1cm}(A) +
 
{\rm ln}\hspace{0.1cm}\left [{\rm exp} (  
 
{\rm ln}\hspace{0.1cm}\left [{\rm exp} (  
 
- \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}) \right ]  
 
- \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}) \right ]  
 
= {\rm ln}\hspace{0.1cm}(A) - \frac{x ^2}{2 \sigma^2}\hspace{0.05cm}.$$
 
= {\rm ln}\hspace{0.1cm}(A) - \frac{x ^2}{2 \sigma^2}\hspace{0.05cm}.$$
<b>b)</b>&nbsp;&nbsp;Mit diesem Ergebnis erhält man für die differentielle Entropie in &bdquo;nat&rdquo;:
+
 
$$h_{\rm nat}(X) \hspace{-0.15cm}  = \hspace{-0.15cm} -\hspace{-0.1cm}  \int_{-\infty}^{+\infty} \hspace{-0.15cm}  f_X(x) \cdot {\rm ln} \hspace{0.1cm} [f_X(x)] \hspace{0.1cm}{\rm d}x$$ $$=\
+
 
  \hspace{-0.15cm}
+
 
 +
'''(2)'''&nbsp; <u>Both proposed solutions</u> are correct:
 +
*Using the result from&nbsp; '''(1)'''&nbsp; we obtain for the differential entropy in&nbsp; "nat":
 +
:$$h_{\rm nat}(X)=  -\hspace{-0.1cm}  \int_{-\infty}^{+\infty} \hspace{-0.15cm}  f_X(x) \cdot {\rm ln} \hspace{0.1cm} [f_X(x)] \hspace{0.1cm}{\rm d}x =
 
- {\rm ln}\hspace{0.1cm}(A) \cdot  
 
- {\rm ln}\hspace{0.1cm}(A) \cdot  
 
\int_{-\infty}^{+\infty} \hspace{-0.15cm}  f_X(x) \hspace{0.1cm}{\rm d}x
 
\int_{-\infty}^{+\infty} \hspace{-0.15cm}  f_X(x) \hspace{0.1cm}{\rm d}x
+ \frac{1}{2 \sigma^2} \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm}  x^2 \cdot f_X(x) \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A)  + \frac{1}{2}
+
+ \frac{1}{2 \sigma^2} \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm}  x^2 \cdot f_X(x) \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A)  + {1}/{2}
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
Hierbei ist berücksichtigt, dass das erste Integral gleich 1 ist (WDF&ndash;Fläche) und das zweite Integral gleich die Varianz <i>&sigma;</i><sup>2</sup> angibt (wenn wie hier der Gleichanteil <i>m</i><sub>1</sub> = 0 ist).
+
*Here it is taken into account that the first integral is equal to&nbsp; $1$&nbsp;&nbsp; (PDF area).
 
+
*The second integral also gives the variance&nbsp; $\sigma^2$&nbsp; (if, as here, the equal part&nbsp; $m_1 = 0$&nbsp;).  
Ersetzt man die Abkürzungsvariable <i>A</i>, so erhält man:
+
*Substituting the abbreviation variable&nbsp; $A$, we obtain:
$$h_{\rm nat}(X) \hspace{-0.15cm}  =  \hspace{-0.15cm}  - {\rm ln}\hspace{0.05cm}\left (\frac{1}{\sqrt{2\pi  \sigma^2}} \right )  + \frac{1}{2} = \frac{1}{2} \cdot {\rm ln}\hspace{0.05cm}\left ({2\pi  \sigma^2} \right ) + \frac{1}{2} \cdot {\rm ln}\hspace{0.05cm}\left ( {\rm e} \right ) = \frac{1}{2} \cdot {\rm ln}\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )
+
:$$h_{\rm nat}(X) \hspace{-0.15cm}  =  \hspace{-0.15cm}  - {\rm ln}\hspace{0.05cm}\left (\frac{1}{\sqrt{2\pi  \sigma^2}} \right )  + {1}/{2} = {1}/{2}\cdot {\rm ln}\hspace{0.05cm}\left ({2\pi  \sigma^2} \right ) + {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ( {\rm e} \right ) = {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
Soll die  differentielle Entropie <i>h</i>(<i>X</i>) nicht in &bdquo;nat&rdquo; angegeben werden, sondern in &bdquo;bit&rdquo;, so ist für den Logarithmus die Basis 2 zu wählen:
+
*If the differential entropy&nbsp; $h(X)$&nbsp; is not to be given in&nbsp; "nat"&nbsp; but in&nbsp; "bit",&nbsp; choose base&nbsp; $2$&nbsp; for the logarithm:
$$h_{\rm bit}(X) = \frac{1}{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )
+
:$$h_{\rm bit}(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
Das heißt: Hier sind <u>beide Lösungsvorschläge</u> richtig.
 
  
<b>c)</b>&nbsp;&nbsp;Nach der impliziten Definition <i>h</i>(<i>X</i>) = 1/2 &middot; log<sub>2</sub> (<i>&Gamma;</i><sub>L</sub> &middot; <i>&sigma;</i><sup>2</sup>) ergibt sich somit für die Kenngröße:
+
 
$${\it \Gamma}_{\rm L} = 2\pi {\rm e} \hspace{0.15cm}\underline{\approx 17.08}
+
 
 +
'''(3)'''&nbsp; Thus, according to the implicit definition&nbsp; $h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2)$&nbsp;, the parameter is:
 +
:$${\it \Gamma}_{\rm L} = 2\pi {\rm e} \hspace{0.15cm}\underline{\approx 17.08}
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
<b>d)</b>&nbsp;&nbsp;Wir betrachten nun eine Gaußsche Wahrscheinlichkeitsdichtefunktion mit Mittelwert <i>m</i><sub>1</sub>:
+
 
$$f_X(x) = \frac{1}{\sqrt{2\pi  \sigma^2}} \cdot {\rm exp}\left [  
+
 
 +
 
 +
'''(4)'''&nbsp; We now consider a Gaussian probability density function with mean&nbsp; $m_1$:
 +
:$$f_X(x) = \frac{1}{\sqrt{2\pi  \sigma^2}} \cdot {\rm exp}\left [  
 
- \hspace{0.05cm}\frac{(x -m_1)^2}{2 \sigma^2} \right ]
 
- \hspace{0.05cm}\frac{(x -m_1)^2}{2 \sigma^2} \right ]
 
  \hspace{0.05cm}.$$
 
  \hspace{0.05cm}.$$
Das zweite Moment <i>m</i><sub>2</sub> = E[|<i>X</i>|<sup>2</sup>] kann man auch als die Leistung <i>P</i> bezeichnen, während für die Varianz gilt: <i>&sigma;</i><sup>2</sup> = E[|<i>X</i> &ndash; <i>m</i><sub>1</sub>|<sup>2</sup>]= <i>&mu;</i><sub>2</sub> (ist gleichzeitig das zweite Zentralmoment)Nach dem Satz von Steiner gilt <i>P</i> = <i>m</i><sub>2</sub> = <i>m</i><sub>1</sub><sup>2</sup> + <i>&sigma;</i><sup>2</sup>. Unter der Voraussetzung <i>m</i><sub>1</sub> = <i>&sigma;</i> = 1 ist somit <i>P</i>/<i>&sigma;</i><sup>2</sup> <u>= 2</u>.
+
* The second moment&nbsp; $m_2 = {\rm E}\big [X ^2 \big ]$&nbsp; can also be called the power&nbsp; $P$,&nbsp; while for the variance holds&nbsp; (this is also the second central moment):
 +
:$$\sigma^2 = {\rm E}\big [|X – m_1|^2 \big ] = \mu_2.$$  
 +
*According to Steiner's theorem,&nbsp; $P = m_2 = m_1^2 + \sigma^2$.&nbsp; Thus, assuming &nbsp;$m_1 = \sigma = 1$ &nbsp; &rArr; &nbsp; $\underline{P/\sigma^2 = 2}$.
  
Durch den Gleichanteil wird zwar die Leistung verdoppelt. An der differentiellen Entropie ändert sich dadurch aber nichts. Es gilt somit
+
*Due to the DC component, the power is indeed doubled.&nbsp; However, this does not change anything in the differential entropy.&nbsp; Thus, it is still valid:
weiterhin:
+
:$$h(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )= {1}/{2} \cdot {\rm log}_2\hspace{0.05cm} (17.08)\hspace{0.15cm}\underline{\approx 2.047\,{\rm bit}}
$$h(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )= {1}/{2} \cdot {\rm log}_2\hspace{0.05cm} (17.08)\hspace{0.15cm}\underline{\approx 2.047\,{\rm bit}}
 
 
\hspace{0.05cm}.$$
 
\hspace{0.05cm}.$$
[[File:P_ID2876__Inf_A_4_3e_neu.png|right|]]
 
<b>e)</b>&nbsp;&nbsp;In der vervollständigten Tabelle sind auch die numerischen Werte der Kenngrößen <i>&Gamma;</i><sub>L</sub> und <i>&Gamma;</i><sub>A</sub> eingetragen.
 
  
Eine Wahrscheinlichkeitsdichtefunktion <i>f<sub>X</sub></i>(<i>x</i>) ist bei  Leistungsbegrenzung immer dann besonders günstig, wenn der Wert <i>&Gamma;</i><sub>L</sub> (rechte Spalte) möglichst groß ist. Dann ist die differentielle Entropie <i>h</i>(<i>X</i>) ebenfalls groß.
 
<br><br>
 
Die numerischen Ergebnisse lassen sich wie folgt interpretieren:
 
:* Wie im [http://en.lntwww.de/Informationstheorie/Differentielle_Entropie#Differentielle_Entropie_einiger_leistungsbegrenzter_Zufallsgr.C3.B6.C3.9Fen  '''Theorieteil''' ] bewiesen, führt die Gaußverteilung <i>f</i><sub>4</sub>(<i>x</i>) hier zum größtmöglichen <i>&Gamma;</i><sub>L</sub> &asymp; 17.08 &#8658;&nbsp; <u>Lösungsvorschlag 1</u> ist richtig (Wert in der letzten Spalte rot markiert).
 
:* Für die Gleichverteilung <i>f</i><sub>1</sub>(<i>x</i>) ist die Kenngröße <i>&Gamma;</i><sub>L</sub> = 12 die kleinste in der gesamten Tabelle &#8658; Lösungsvorschlag 2 ist falsch.
 
:* Die Dreieckverteilung <i>f</i><sub>2</sub>(<i>x</i>)  ist mit <i>&Gamma;</i><sub>L</sub> = 16.31  günstiger als die Gleichverteilung und auch besser als die Laplaceverteilung (<i>f</i><sub>3</sub>(<i>x</i>, <i>&Gamma;</i><sub>L</sub> = 14.78) &#8658; auch Lösungsvorschlag 3 ist falsch.
 
  
<b>f)</b>&nbsp;&nbsp; Eine WDF <i>f<sub>X</sub></i>(<i>x</i>) ist unter der Nebenbedingung der Spitzenwertbegrenzung &nbsp;&#8658;&nbsp; |<i>X</i>|&nbsp;&#8804;&nbsp;<i>A</i> günstig hinsichtlich der differentiellen Entropie <i>h</i>(<i>X</i>), wenn <i>&Gamma;</i><sub>A</sub> (mittlere Spalte) möglichst groß ist:
+
[[File:EN_Inf_A_4_3_M_v2L.png|right|frame|Completed results table for&nbsp; $h(X)$]]
:* Wie im [http://en.lntwww.de/Informationstheorie/Differentielle_Entropie#Differentielle_Entropie_einiger_spitzenwertbegrenzter_Zufallsgr.C3.B6.C3.9Fen '''Theorieteil'''] gezeigt wird, führt die Gleichverteilung <i>f</i><sub>1</sub>(<i>x</i>) hier zum größtmöglichen <i>&Gamma;</i><sub>A</sub> = 2  &#8658;&nbsp; <u>Lösungsvorschlag 2</u> ist richtig (Wert in der mittleren Spalte rot markiert).
+
'''(5)'''&nbsp; Correct are the proposed solutions&nbsp; '''(1)'''&nbsp; and&nbsp; '''(4)'''.&nbsp; The numerical values of the characteristics&nbsp; ${\it \Gamma}_{\rm L}$&nbsp; and&nbsp; ${\it \Gamma}_{\rm A}$&nbsp; are also entered in the completed table on the right.
:* Die ebenfalls spitzenwertbegrenzte Dreieckverteilung <i>f</i><sub>2</sub>(<i>x</i>) ist durch ein etwas kleineres <nobr><i>&Gamma;</i><sub>A</sub> = 1.649</nobr> gekennzeichnet
+
 
&nbsp;&#8658;&nbsp; Lösungsvorschlag 3 ist falsch.  
+
A probability density function&nbsp; $f_X(x)$&nbsp; is always particularly favorable under power constraints if the value&nbsp; ${\it \Gamma}_{\rm L}$&nbsp; (right column)&nbsp; is as large as possible.&nbsp; Then the differential entropy&nbsp; $h(X)$&nbsp; is also large.
:* Die Gaußverteilung <i>f</i><sub>4</sub>(<i>x</i>) ist unendlich weit ausgedehnt. Eine Spitzenwertbegrenzung auf |<i>X</i>| &#8804; <i>A</i> führt hier zu Diracfunktionen in der WDF &nbsp;&#8658;&nbsp; <i>h</i>(<i>X</i>)&nbsp;=&nbsp;&ndash;&#8734;, siehe [http://en.lntwww.de/Aufgaben:4.02Z_Gemischte_Zufallsgr%C3%B6%C3%9Fen#collapse1 '''Aufgabe Z4.2 &ndash; Musterlösung.''']
+
 
:* Gleiches würde auch für die Laplaceverteilung <i>f</i><sub>3</sub>(<i>x</i>) gelten.
+
The numerical results can be interpreted as follows:
 +
* As is proved in the theory part, the Gaussian PDF&nbsp; $f_4(x)$&nbsp; leads here to the largest possible&nbsp; ${\it \Gamma}_{\rm L} &asymp; 17.08$ &nbsp; &#8658; &nbsp; the <u>proposed solution 1</u> is correct (the value in the last column is marked in red).
 +
* For the uniform PDF&nbsp; $f_1(x)$&nbsp; the parameter ${\it \Gamma}_{\rm L} = 12$&nbsp; is the smallest in the whole table &nbsp; &#8658; &nbsp; the proposed solution 2 is wrong.
 +
* The triangular PDF&nbsp; $f_2(x)$&nbsp; with&nbsp; ${\it \Gamma}_{\rm L} = 16.31$&nbsp;  is more favorable than the uniform PDF &nbsp; &#8658; &nbsp; the proposed solution 3 is wrong.
 +
*The triangular PDF&nbsp; $f_2(x)$&nbsp; is also better than the Laplace PDF&nbsp; $f_2(x) \ \ ({\it \Gamma}_{\rm L} = 14.78)$ &nbsp; &#8658; &nbsp; the <u>proposed solution 4</u> is correct.
 +
 
 +
 
 +
 
 +
 
 +
'''(6)'''&nbsp; Correct is the proposed solution&nbsp; '''(2)'''.&nbsp; A PDF&nbsp; $f_X(x)$&nbsp; is favorable in terms of differential entropy&nbsp; $h(X)$ under the peak constraint &nbsp; &#8658; &nbsp;  $|X| ≤ A$,&nbsp; if the weighting factor&nbsp;  ${\it \Gamma}_{\rm A}$&nbsp; (middle column)&nbsp; is as large as possible:
 +
* As shown in the theory section, the uniform distribution &nbsp; $f_1(x)$&nbsp; leads here to the largest possible&nbsp; ${\it \Gamma}_{\rm A}= 2$  &nbsp; &#8658; &nbsp; the <u>proposed solution 2</u> is correct&nbsp; (the value in the middle column is marked in red).
 +
* The triangular PDF&nbsp; $f_2(x)$,&nbsp; which is also peak-constrained, is characterized by a somewhat smaller&nbsp; ${\it \Gamma}_{\rm A}= 1.649$&nbsp; &nbsp; &#8658; &nbsp; dthe proposed solution 3 is incorrect.
 +
* The Gaussian PDF&nbsp; $f_4(x)$&nbsp; is infinitely extended.&nbsp; A peak constraint on&nbsp; $|X| ≤ A$&nbsp; leads here to Dirac functions in the PDF &nbsp; &#8658; &nbsp; $h(X) \to - \infty$, see sample solution to Exercise 4.2Z, subtask '''(4)'''.
 +
* The same would be true for the Laplace PDF&nbsp; $f_3(x)$&nbsp;.
  
 
{{ML-Fuß}}
 
{{ML-Fuß}}
Line 133: Line 163:
  
  
[[Category:Aufgaben zu Informationstheorie|^4.1  Differentielle Entropie^]]
+
[[Category:Information Theory: Exercises|^4.1  Differential Entropy^]]

Latest revision as of 14:08, 28 September 2021

$h(X)$  for four probability density functions

The adjacent table shows the comparison result with respect to the differential entropy  $h(X)$  for

$$f_1(x) = \left\{ \begin{array}{c} 1/(2A) \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\ {\rm else} \\ \end{array} ,$$
$$f_2(x) = \left\{ \begin{array}{c} 1/A \cdot \big [1 - |x|/A \big ] \\ 0 \\ \end{array} \right. \begin{array}{*{20}c} {\rm{f\ddot{u}r}} \hspace{0.1cm} |x| \le A \\ {\rm else} \\ \end{array} ,$$
$$f_3(x) = \lambda/2 \cdot {\rm e}^{-\lambda \hspace{0.05cm} \cdot \hspace{0.05cm}|x|}\hspace{0.05cm}.$$

The values for the  Gaussian distribution   ⇒   $f_X(x) = f_4(x)$  with

$$f_4(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \cdot {\rm e}^{ - \hspace{0.05cm}{x ^2}/{(2 \sigma^2})}$$

are not yet entered here.  These are to be determined in subtasks  (1)  to  (3) .

Each probability density function  $\rm (PDF)$  considered here is

  • symmetric about  $x = 0$    ⇒   $f_X(-x) = f_X(x)$
  • and thus zero mean   ⇒  $m_1 = 0$.


In all cases considered here, the differential entropy can be represented as follows:

  • Under the constraint  $|X| ≤ A$   ⇒    peak constraint  $($German:  "Spitzenwertbegrenzung"  or  "Amplitudenbegrenzung"   ⇒   Identifier:  $\rm A)$:
$$h(X) = {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm A} \cdot A) \hspace{0.05cm},$$
  • Under the constraint  ${\rm E}\big [|X – m_1|^2 \big ] ≤ σ^2$   ⇒   power constraint  $($German:  "Leistungsbegrenzung"   ⇒   Identifier:  $\rm L)$:
$$h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2) \hspace{0.05cm}.$$

The larger the respective parameter  ${\it \Gamma}_{\hspace{-0.01cm}\rm A}$  or  ${\it \Gamma}_{\hspace{-0.01cm}\rm L}$  is, the more favorable is the present PDF in terms of differential entropy for the agreed constraint.





Hints:

  • The exercise belongs to the chapter  Differential Entropy.
  • Useful hints for solving this task can be found in particular on the pages
Differential entropy of some peak-constrained random variables
Differential entropy of some power-constrained random variables.



Questions

1

Which equation is valid for the logarithm of the Gaussian PDF?

It holds:   $\ln \big[f_X(x) \big] = \ln (A) - x^2/(2 \sigma^2)$   with   $A = f_X(x=0)$.
Es It holds:   $\ln \big [f_X(x) \big] = A - \ln (x^2/(2 \sigma^2)$   with   $A = f_X(x=0)$.

2

Which equation holds for the differential entropy of the Gaussian PDF?

It holds:   $h(X)= 1/2 \cdot \ln (2\pi\hspace{0.05cm}{\rm e}\hspace{0.01cm}\cdot\hspace{0.01cm}\sigma^2)$  with the pseudo-unit  "nat".
It holds:   $h(X)= 1/2 \cdot \log_2 (2\pi\hspace{0.05cm}{\rm e}\hspace{0.01cm}\cdot\hspace{0.01cm}\sigma^2)$  with the pseudo-unit  "bit".

3

Complete the missing entry for the Gaussian PDF in the above table.

${\it \Gamma}_{\rm L} \ = \ $

4

What values are obtained for the Gaussian PDF with the DC component  $m_1 = \sigma = 1$?

$P/\sigma^2 \ = \ $

$h(X) \ = \ $

$\ \rm bit$

5

Which of the statements are true for the differential entropy  $h(X)$  considering the  "power constraint"  ${\rm E}\big[|X – m_1|^2\big] ≤ σ^2$?

The Gaussian PDF   ⇒   $f_4(x)$  leads to the maximum  $h(X)$.
The uniform PDF   ⇒   $f_1(x)$  leads to the maximum  $h(X)$.
The triangular PDF   ⇒   $f_2(x)$  is very unfavorable because it is peak-constrained.
The triangular PDF   ⇒   $f_2(x)$  is more favorable than the Laplace PDF   ⇒   $f_3(x)$.

6

Which of the statements are true for  "peak constraint"  to the range  $|X| ≤ A$.  The maximum differential entropy  $h(X)$  is obtained for

a Gaussian PDF   ⇒   $f_4(x)$  followed by a constraint   ⇒  $|X| ≤ A$,
the uniform PDF   ⇒   $f_1(x)$,
the triangular PDF   ⇒   $f_2(x)$.


Solution

(1)  We assume the zero mean Gaussian PDF:

$$f_X(x) = f_4(x) =A \cdot {\rm exp} [ - \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}] \hspace{0.5cm}{\rm with}\hspace{0.5cm} A = \frac{1}{\sqrt{2\pi \sigma^2}}\hspace{0.05cm}.$$
  • Logarithmizing this function, the result is proposed solution 1:
$${\rm ln}\hspace{0.1cm} \big [f_X(x) \big ] = {\rm ln}\hspace{0.1cm}(A) + {\rm ln}\hspace{0.1cm}\left [{\rm exp} ( - \hspace{0.05cm}\frac{x ^2}{2 \sigma^2}) \right ] = {\rm ln}\hspace{0.1cm}(A) - \frac{x ^2}{2 \sigma^2}\hspace{0.05cm}.$$


(2)  Both proposed solutions are correct:

  • Using the result from  (1)  we obtain for the differential entropy in  "nat":
$$h_{\rm nat}(X)= -\hspace{-0.1cm} \int_{-\infty}^{+\infty} \hspace{-0.15cm} f_X(x) \cdot {\rm ln} \hspace{0.1cm} [f_X(x)] \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A) \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm} f_X(x) \hspace{0.1cm}{\rm d}x + \frac{1}{2 \sigma^2} \cdot \int_{-\infty}^{+\infty} \hspace{-0.15cm} x^2 \cdot f_X(x) \hspace{0.1cm}{\rm d}x = - {\rm ln}\hspace{0.1cm}(A) + {1}/{2} \hspace{0.05cm}.$$
  • Here it is taken into account that the first integral is equal to  $1$   (PDF area).
  • The second integral also gives the variance  $\sigma^2$  (if, as here, the equal part  $m_1 = 0$ ).
  • Substituting the abbreviation variable  $A$, we obtain:
$$h_{\rm nat}(X) \hspace{-0.15cm} = \hspace{-0.15cm} - {\rm ln}\hspace{0.05cm}\left (\frac{1}{\sqrt{2\pi \sigma^2}} \right ) + {1}/{2} = {1}/{2}\cdot {\rm ln}\hspace{0.05cm}\left ({2\pi \sigma^2} \right ) + {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ( {\rm e} \right ) = {1}/{2} \cdot {\rm ln}\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right ) \hspace{0.05cm}.$$
  • If the differential entropy  $h(X)$  is not to be given in  "nat"  but in  "bit",  choose base  $2$  for the logarithm:
$$h_{\rm bit}(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right ) \hspace{0.05cm}.$$


(3)  Thus, according to the implicit definition  $h(X) = {1}/{2} \cdot {\rm log} \hspace{0.1cm} ({\it \Gamma}_{\hspace{-0.01cm}\rm L} \cdot \sigma^2)$ , the parameter is:

$${\it \Gamma}_{\rm L} = 2\pi {\rm e} \hspace{0.15cm}\underline{\approx 17.08} \hspace{0.05cm}.$$


(4)  We now consider a Gaussian probability density function with mean  $m_1$:

$$f_X(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \cdot {\rm exp}\left [ - \hspace{0.05cm}\frac{(x -m_1)^2}{2 \sigma^2} \right ] \hspace{0.05cm}.$$
  • The second moment  $m_2 = {\rm E}\big [X ^2 \big ]$  can also be called the power  $P$,  while for the variance holds  (this is also the second central moment):
$$\sigma^2 = {\rm E}\big [|X – m_1|^2 \big ] = \mu_2.$$
  • According to Steiner's theorem,  $P = m_2 = m_1^2 + \sigma^2$.  Thus, assuming  $m_1 = \sigma = 1$   ⇒   $\underline{P/\sigma^2 = 2}$.
  • Due to the DC component, the power is indeed doubled.  However, this does not change anything in the differential entropy.  Thus, it is still valid:
$$h(X) = {1}/{2} \cdot {\rm log}_2\hspace{0.05cm}\left ({{2\pi {\rm e} \cdot \sigma^2}} \right )= {1}/{2} \cdot {\rm log}_2\hspace{0.05cm} (17.08)\hspace{0.15cm}\underline{\approx 2.047\,{\rm bit}} \hspace{0.05cm}.$$


Completed results table for  $h(X)$

(5)  Correct are the proposed solutions  (1)  and  (4).  The numerical values of the characteristics  ${\it \Gamma}_{\rm L}$  and  ${\it \Gamma}_{\rm A}$  are also entered in the completed table on the right.

A probability density function  $f_X(x)$  is always particularly favorable under power constraints if the value  ${\it \Gamma}_{\rm L}$  (right column)  is as large as possible.  Then the differential entropy  $h(X)$  is also large.

The numerical results can be interpreted as follows:

  • As is proved in the theory part, the Gaussian PDF  $f_4(x)$  leads here to the largest possible  ${\it \Gamma}_{\rm L} ≈ 17.08$   ⇒   the proposed solution 1 is correct (the value in the last column is marked in red).
  • For the uniform PDF  $f_1(x)$  the parameter ${\it \Gamma}_{\rm L} = 12$  is the smallest in the whole table   ⇒   the proposed solution 2 is wrong.
  • The triangular PDF  $f_2(x)$  with  ${\it \Gamma}_{\rm L} = 16.31$  is more favorable than the uniform PDF   ⇒   the proposed solution 3 is wrong.
  • The triangular PDF  $f_2(x)$  is also better than the Laplace PDF  $f_2(x) \ \ ({\it \Gamma}_{\rm L} = 14.78)$   ⇒   the proposed solution 4 is correct.



(6)  Correct is the proposed solution  (2).  A PDF  $f_X(x)$  is favorable in terms of differential entropy  $h(X)$ under the peak constraint   ⇒   $|X| ≤ A$,  if the weighting factor  ${\it \Gamma}_{\rm A}$  (middle column)  is as large as possible:

  • As shown in the theory section, the uniform distribution   $f_1(x)$  leads here to the largest possible  ${\it \Gamma}_{\rm A}= 2$   ⇒   the proposed solution 2 is correct  (the value in the middle column is marked in red).
  • The triangular PDF  $f_2(x)$,  which is also peak-constrained, is characterized by a somewhat smaller  ${\it \Gamma}_{\rm A}= 1.649$    ⇒   dthe proposed solution 3 is incorrect.
  • The Gaussian PDF  $f_4(x)$  is infinitely extended.  A peak constraint on  $|X| ≤ A$  leads here to Dirac functions in the PDF   ⇒   $h(X) \to - \infty$, see sample solution to Exercise 4.2Z, subtask (4).
  • The same would be true for the Laplace PDF  $f_3(x)$ .