We consider the two continuous-valued random variables $X$ and $Y$ with probability density functions $f_X(x)$ and $f_Y(y)$. For these random variables one can
We consider the two continuous random variables $X$ and $Y$ with probability density functions $f_X(x)$ and $f_Y(y)$. For these random variables one can
* not specify the conventional entropies $H(X)$ and $H(Y)$ , respectively,
* not specify the conventional entropies $H(X)$ and $H(Y)$ , respectively,
* but the differential entropies $h(X)$ and $h(Y)$.
* but the differential entropies $h(X)$ and $h(Y)$.
We also consider two discrete value random variables:
We also consider two discrete random variables:
*The random variable $Z_{X,\hspace{0.05cm}M}$ is obtained by (suitably) quantizing the random variable $X$ with the quantization step number $M$ <br>⇒ quantization interval width ${\it \Delta} = 0.5/M$.
*The variable $Z_{X,\hspace{0.05cm}M}$ is obtained by (suitably) quantizing the random quantity $X$ with the quantization level number $M$ <br>⇒ quantization interval width ${\it \Delta} = 0.5/M$.
* The random quantity $Z_{Y,\hspace{0.05cm}M}$ is obtained after quantization of the random quantity $Y$ with the quantization level number $M$ <br>⇒ quantization interval width ${\it \Delta} = 2/M$.
* The variable $Z_{Y,\hspace{0.05cm}M}$ is obtained after quantization of the random quantity $Y$ with the quantization level number $M$ <br>⇒ quantization interval width ${\it \Delta} = 2/M$.
The probability density functions of these discrete random variables are each composed of $M$ Dirac functions whose momentum weights are given by the interval areas of the associated continuous-value random variables.
The probability density functions $\rm (PDF)$ of these discrete random variables are each composed of $M$ Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.
From this, the entropies $H(Z_{X,\hspace{0.05cm}M})$ and $H(Z_{Y,\hspace{0.05cm}M})$ can be determined in the conventional way according to the chapter [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy|Probability Mass Function and Entropy]] .
From this, the entropies $H(Z_{X,\hspace{0.05cm}M})$ and $H(Z_{Y,\hspace{0.05cm}M})$ can be determined in the conventional way according to the section [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Probability_mass_function_and_entropy|Probability mass function and entropy]] .
In the section [[Information_Theory/Differentielle_Entropie#Entropy_of_value-continuous_random_variables_after_quantization|Entropy of Continuous-Value Random Variables after Quantization]] , an approximation was also given. For example:
In the section [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization|Entropy of value-ontinuous random variables after quantization]], an approximation was also given. For example:
*In the course of the task it will be shown that in the case of rectangular PDF ⇒ uniform distribution this "approximation" gives the same result as the direct calculation.
*In the course of the task it will be shown that in the case of rectangular PDF ⇒ uniform distribution this "approximation" gives the same result as the direct calculation.
*But in the general case – so in [[Information_Theory/Differentielle_Entropie#Entropy_of_value-continuous_random_variables_after_quantization|$\text{Example 2}$]] with triangular PDF – above equation is in fact only an approximation, which agrees with the actual entropy $H(Z_{X,\hspace{0.05cm}M})$ only in the limiting case ${\it \Delta} \to 0$ .
*But in the general case – so in [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization|$\text{Example 2}$]] with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy $H(Z_{X,\hspace{0.05cm}M})$ only in the limiting case ${\it \Delta} \to 0$ .
Line 33:
Line 33:
Hints:
Hints:
*The exercise belongs to the chapter [[Information_Theory/Differentielle_Entropie|Differential Entropy]].
*The exercise belongs to the chapter [[Information_Theory/Differentielle_Entropie|Differential Entropy]].
*Useful hints for solving this task can be found in particular on the page [[Information_Theory/Differentielle_Entropie#Entropy_of_value-continuous_random_variables_after_quantization|Entropy of continuous-valued random variables after quantization]] .
*Useful hints for solving this task can be found in particular in the section [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization|Entropy of continuous random variables after quantization]].
===Fragebogen===
===Questions===
<quiz display=simple>
<quiz display=simple>
{Berechnen Sie die differentielle Entropie $h(X)$.
{Calculate the differential entropy $h(X)$.
|type="{}"}
|type="{}"}
$ h(X) \ = \ $ { -1.03--0.97 } $\ \rm bit$
$ h(X) \ = \ $ { -1.03--0.97 } $\ \rm bit$
{Berechnen Sie die differentielle Entropie $h(Y)$.
{Calculate the differential entropy $h(Y)$.
|type="{}"}
|type="{}"}
$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$
$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$
{Berechnen Sie die Entropie der wertdiskreten Zufallsgrößen $Z_{X,\hspace{0.05cm}M=4}$ nach der direkten Methode</u>.
{Calculate the entropy of the discrete random variables $Z_{X,\hspace{0.05cm}M=4}$ using the direct method</u>.
'''(2)''' Mit $y_{\rm min} = -1$ und $y_{\rm max} = +1$ ergibt sich dagegen für die differentielle Entropie der Zufallsgröße $Y$:
'''(2)''' On the other hand, with $y_{\rm min} = -1$ and $y_{\rm max} = +1$ the differential entropy of the random variable $Y$ is given by:
[[File:P_ID2879__Inf_A_4_4c.png|right|frame|Quantisierte Zufallsgröße $Z_{X, \ M = 4}$]]
[[File:P_ID2879__Inf_A_4_4c.png|right|frame|Quantized random variable $Z_{X, \ M = 4}$]]
'''(3)''' Die nebenstehende Grafik verdeutlicht die bestmögliche Quantisierung der Zufallsgröße $X$ mit der Quantisierungsstufenzahl $M = 4$ ⇒ Zufallsgröße $Z_{X, \ M = 4}$:
'''(3)''' The adjacent graph illustrates the best possible quantization of random variable $X$ with quantization level number $M = 4$ ⇒ random variable $Z_{X, \ M = 4}$:
*Die Intervallbreite ist hier gleich ${\it \Delta} = 0.5/4 = 1/8$.
*The interval width here is equal to ${\it \Delta} = 0.5/4 = 1/8$.
*Die möglichen Werte (jeweils in der Intervallmitte) sind $z \in \{0.0625,\ 0.1875,\ 0.3125,\ 0.4375\}$.
*The possible values (at the center of the interval, respectively) are $z \in \{0.0625,\ 0.1875,\ 0.3125,\ 0.4375\}$.
Die <u>direkte Entropieberechnung</u> ergibt mit der Wahrscheinlichkeitsfunktion $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:
Using the probability mass function, the <u>direct entropy calculation</u> gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:
<i>Note:</i> Only in the case of uniform distribution, the approximation gives exactly the same result as the direct calculation, i.e. the actual entropy.
<i>Hinweis:</i> Nur bei der Gleichverteilung liefert die Näherung genau das gleiche Ergebnis wie die direkte Berechnung, also die tatsächliche Entropie.
[[File:P_ID2880__Inf_A_4_4d.png|right|frame|Quantisierte Zufallsgröße $Z_{Y, \ M = 4}$]]
[[File:P_ID2880__Inf_A_4_4d.png|right|frame|Quantized random variable $Z_{Y, \ M = 4}$]]
<br>
<br>
'''(4)''' Aus der zweiten Grafik erkennt man die Gemeinsamkeiten / Unterschiede zur Teilaufgabe '''(3)''':
'''(4)''' From the second graph, one can see the similarities / differences to subtask '''(3)''':
* Der Quantisierungsparameter ist nun ${\it \Delta} = 2/4 = 1/2$.
* The quantization parameter is now ${\it \Delta} = 2/4 = 1/2$.
* Die möglichen Werte sind nun $z \in \{\pm 0.75,\ \pm 0.25\}$.
* The possible values are now $z \in \{\pm 0.75,\ \pm 0.25\}$.
* Somit liefert hier die "Näherung" (ebenso wie die direkte Berechnung) das Ergebnis:
* Thus, here the "approximation" (as well as the direct calculation) gives the result:
[[File:P_ID2881__Inf_A_4_4e.png|right|frame|Quantisierte Zufallsgröße $Z_{Y, \ M = 8}$]]
[[File:P_ID2881__Inf_A_4_4e.png|right|frame|Quantized random variable $Z_{Y, \ M = 8}$]]
'''(5)''' Im Gegensatz zur Teilaufgabe '''(5)''' gilt nun ${\it \Delta} = 1/4$. Daraus folgt für die "Näherung":
'''(5)''' In contrast to subtask '''(5)''': ${\it \Delta} = 1/4$ is now valid. From this follows for the "approximation":
Again, one gets the same result as in the direct calculation.
Man erhält wieder das gleiche Ergebnis wie bei der direkten Berechnung.
<br clear=all>
'''(6)''' Richtig ist nur die <u>Aussage 1</u>:
* Die Entropie $H(Z)$ einer diskreten Zufallsgröße $Z = \{z_1, \ \text{...} \ , z_M\}$ ist nie negativ.
*Der Grenzfall $H(Z) = 0$ ergibt sich zum Beispiel für ${\rm Pr}(Z = z_1) = 1$ und ${\rm Pr}(Z = z_\mu) = 0$ für $2 \le \mu \le M$.
* Dagegen kann die differentielle Entropie $h(X)$ einer wertkontinuierlichen Zufallsgröße $X$ wie folgt sein:
** $h(X) < 0$ $($Teilaufgabe 1$)$,
'''(6)''' Only <u>statement 1</u> is correct:
** $h(X) > 0$ $($Teilaufgabe 2$)$, oder auch
* The entropy $H(Z)$ of a discrete random variable $Z = \{z_1, \ \text{...} \ , z_M\}$ is never negative.
**$h(X) = 0$ $($zum Beispiel für $x_{\rm min} = 0$ und $x_{\rm max} = 1)$.
*For example, the limiting case $H(Z) = 0$ results for ${\rm Pr}(Z = z_1) = 1$ and ${\rm Pr}(Z = z_\mu) = 0$ for $2 \le \mu \le M$.
* In contrast, the differential entropy $h(X)$ of a continuous value random variable $X$ can be as follows:
** $h(X) < 0$ $($subtask 1$)$,
** $h(X) > 0$ $($subtask 2$)$, or even
**$h(X) = 0$ $($for example fo $x_{\rm min} = 0$ and $x_{\rm max} = 1)$.
We consider the two continuous random variables $X$ and $Y$ with probability density functions $f_X(x)$ and $f_Y(y)$. For these random variables one can
not specify the conventional entropies $H(X)$ and $H(Y)$ , respectively,
but the differential entropies $h(X)$ and $h(Y)$.
We also consider two discrete random variables:
The variable $Z_{X,\hspace{0.05cm}M}$ is obtained by (suitably) quantizing the random quantity $X$ with the quantization level number $M$ ⇒ quantization interval width ${\it \Delta} = 0.5/M$.
The variable $Z_{Y,\hspace{0.05cm}M}$ is obtained after quantization of the random quantity $Y$ with the quantization level number $M$ ⇒ quantization interval width ${\it \Delta} = 2/M$.
The probability density functions $\rm (PDF)$ of these discrete random variables are each composed of $M$ Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.
From this, the entropies $H(Z_{X,\hspace{0.05cm}M})$ and $H(Z_{Y,\hspace{0.05cm}M})$ can be determined in the conventional way according to the section Probability mass function and entropy .
In the course of the task it will be shown that in the case of rectangular PDF ⇒ uniform distribution this "approximation" gives the same result as the direct calculation.
But in the general case – so in $\text{Example 2}$ with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy $H(Z_{X,\hspace{0.05cm}M})$ only in the limiting case ${\it \Delta} \to 0$ .
(3) The adjacent graph illustrates the best possible quantization of random variable $X$ with quantization level number $M = 4$ ⇒ random variable $Z_{X, \ M = 4}$:
The interval width here is equal to ${\it \Delta} = 0.5/4 = 1/8$.
The possible values (at the center of the interval, respectively) are $z \in \{0.0625,\ 0.1875,\ 0.3125,\ 0.4375\}$.
Using the probability mass function, the direct entropy calculation gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:
