Difference between revisions of "Aufgaben:Exercise 3.2: Expected Value Calculations"

From LNTwww
 
(21 intermediate revisions by 4 users not shown)
Line 1: Line 1:
  
{{quiz-Header|Buchseite=Informationstheorie/Einige Vorbemerkungen zu zweidimensionalen Zufallsgrößen
+
{{quiz-Header|Buchseite=Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables
 
}}
 
}}
  
[[File:P_ID2751__Inf_A_3_2.png|right|]]
+
[[File:P_ID2751__Inf_A_3_2.png|right|frame|Two-dimensional <br>probability mass function]]
Wir betrachten folgende Wahrscheinlichkeitsfunktionen:
+
We consider the following probability mass functions&nbsp; $\rm (PMF)$:
  
:* <i>P<sub>X</sub></i>(<i>X</i>) = [1/2, 1/8, 0, 3/8],
+
:$$P_X(X) = \big[1/2,\ 1/8,\ 0,\ 3/8 \big],$$
 +
:$$P_Y(Y) = \big[1/2,\ 1/4,\ 1/4,\ 0 \big],$$
 +
:$$P_U(U) = \big[1/2,\ 1/2 \big],$$
 +
:$$P_V(V) = \big[3/4,\ 1/4\big].$$
  
:* <i>P<sub>Y</sub></i>(<i>Y</i>) = [1/2, 1/4, 1/4, 0],
+
For the associated random variables, let:
  
:* <i>P<sub>U</sub></i>(<i>U</i>) = [1/2, 1/2],
+
: $X= \{0,\ 1,\ 2,\ 3\}$, &nbsp; &nbsp;  $Y= \{0,\ 1,\ 2,\ 3\}$,&nbsp; &nbsp; $U = \{0,\ 1\}$, &nbsp; &nbsp; $V = \{0, 1\}$.
  
:* <i>P<sub>V</sub></i>(<i>V</i>) = [3/4, 1/4].
+
Often, for such discrete random variables, one must have to calculate different expected values of the form
 +
:$${\rm E} \big [ F(X)\big ] =\hspace{-0.3cm}  \sum_{x \hspace{0.05cm}\in \hspace{0.05cm}\hspace{-0.03cm}  {\rm supp} (P_X)}  \hspace{-0.1cm}
 +
P_{X}(x) \cdot F(x). $$
  
Für die dazugehörigen Zufallsgrößen gelte:
+
Here, denote:
  
:* <i>X</i> = {0, 1, 2, 3}, <i>Y</i> = {0, 1, 2, 3},
+
* $P_X(X)$&nbsp; denotes the probability mass function of the discrete random variable &nbsp; $X$.
 +
* The&nbsp; "support"&nbsp; of&nbsp; $P_X$&nbsp; includes all those realisations&nbsp; $x$&nbsp; of the random variable&nbsp; $X$&nbsp; with non-vanishing probability.
 +
*Formally, this can be written as
 +
:$${\rm supp} (P_X)  = \{ x: \hspace{0.25cm}x \in X \hspace{0.15cm}\underline{\rm and} \hspace{0.15cm} P_X(x) \ne 0 \} \hspace{0.05cm}.$$
 +
* $F(X)$&nbsp; is an (arbitrary) real-valued function that can be specified in the entire domain of definition of the random variable&nbsp; $X$&nbsp;.
  
:* <i>U</i> = {0, 1}, <i>V</i> = {0, 1}.
 
  
Oft muss man in der Informationstheorie für solche diskreten Zufallsgrößen verschiedene Erwartungswerte der Form
+
In the task, the expected values for various functions&nbsp; $F(X)$&nbsp; are to be calculated, among others for
:$${\rm E} \left [ F(X)\right ] = \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} {\rm supp} (P_X)}  \hspace{-0.1cm}
 
P_{X}(x) \cdot F(x) $$
 
  
berechnen. Hierbei bedeuten:
+
#&nbsp; $F(X)= 1/P_X(X)$,
 +
#&nbsp; $F(X)= P_X(X)$,
 +
#&nbsp; $F(X)= - \log_2 \ P_X(X)$.
  
:* <i>P<sub>X</sub></i>(<i>X</i>) bezeichnet die <i>Wahrscheinlichkeitsfunktion</i> der diskreten Zufallsgröße <i>X</i>.
 
  
:* Der <i>Support</i> von <i>P<sub>x</sub></i> umfasst alle diejenigen Realisierungen <i>x</i> der Zufallsgröße <i>X</i> mit nicht verschwindender Wahrscheinlichkeit. Formal kann hierfür geschrieben werden:
 
:$${\rm supp} (P_X)  = \{ x: \hspace{0.25cm}x \in X \hspace{0.15cm}\underline{\rm und} \hspace{0.15cm} P_X(x) \ne 0 \} \hspace{0.05cm}.$$
 
  
:* <i>F</i>(<i>X</i>) ist eine (beliebige) reellwertige Funktion, die im gesamten Definitionsgebiet der Zufallsgröße angebbar ist.
 
  
In der Aufgabe sollen die Erwartungswerte für verschiedene Funktionen <i>F</i>(<i>X</i>) berechnet werden, unter anderem für
 
  
:* <i>F</i>(<i>X</i>) = 1/<i>P<sub>X</sub></i>(<i>X</i>),
 
  
:* <i>F</i>(<i>X</i>) = <i>P<sub>X</sub></i>(<i>X</i>),
 
  
:* <i>F</i>(<i>X</i>) = &ndash;log<sub>2</sub> <i>P<sub>X</sub></i>(<i>X</i>).
 
  
<b>Hinweis:</b> Die Aufgabe gehört zum Themengebiet von Kapitel 3.1 dieses Buches. Weiter gilt, ohne dass dies für die Lösung dieser Aufgabe von Interesse ist:
+
Hints:  
 +
*The exercise belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on two-dimensional  random variables]].
 +
* The two one-dimensional  probability mass functions&nbsp; $P_X(X)$&nbsp; and&nbsp; $P_Y(Y)$&nbsp; result from the presented 2D&ndash;PMF&nbsp; $P_{XY}(X,\ Y)$,&nbsp; as will be shown in&nbsp; [[Aufgaben:3.2Z_2D–Wahrscheinlichkeitsfunktion|Exercise 3.2Z]].
 +
* The binary probability mass functions&nbsp; $P_U(U)$&nbsp; and&nbsp; $P_V(V)$&nbsp; are obtained according to the modulo operations&nbsp; $U = X \hspace{0.1cm}\text{mod} \hspace{0.1cm}2$&nbsp; and&nbsp; $V = Y \hspace{0.1cm}\text{mod} \hspace{0.1cm} 2$.
 +
  
:* Die beiden &bdquo;eindimensionalen&rdquo; Wahrscheinlichkeitsfunktionen <i>P<sub>X</sub></i>(<i>X</i>) und <i>P<sub>Y</sub></i>(<i>Y</i>) ergeben sich aus der dargestellten 2D&ndash;PMF <i>P<sub>XY</sub></i>(<i>X</i>, <i>Y</i>), wie in Aufgabe Z3.2 gezeigt werden soll.
 
  
:* Zu den binären Wahrscheinlichkeitsfunktionen <i>P<sub>U</sub></i>(<i>U</i>) und <i>P<sub>V</sub></i>(<i>V</i>) kommt man entsprechend den Modulo&ndash;Operationen <i>U</i> = <i>X</i> mod 2 sowie <i>V</i> = <i>Y</i> mod 2 (siehe ebenfalls Aufgabe Z3.2)
+
===Questions===
 
 
 
 
===Fragebogen===
 
  
 
<quiz display=simple>
 
<quiz display=simple>
{Welche Ergebnisse liefern die folgenden Erwartungswerte?
+
{What are the results of the following expected values?
 
|type="{}"}
 
|type="{}"}
$E[1/P_X(X)]$ = { 3 3% }
+
${\rm E}\big[1/P_X(X)\big] \ = \ $ { 3 3% }
$E[1/P_Y(Y)]$ = { 3 3% }
+
${\rm E}\big[1/P_{\hspace{0.04cm}Y}(\hspace{0.02cm}Y\hspace{0.02cm})\big] \ = \ ${ 3 3% }
  
  
{Geben Sie die folgenden Erwartungswerte an:
+
{Give the following expected values:
 
|type="{}"}
 
|type="{}"}
$E[P_X(X)]$ = { 0.406 3% }
+
${\rm E}\big[P_X(X)\big] \ = \ $ { 0.406 3% }
$E[P_Y(Y)]$ = { 0.375 3% }
+
${\rm E}\big[P_Y(Y)\big] \ = \ $ { 0.375 3% }
  
  
{Berechnen Sie nun die folgenden Erwartungswerte:
+
{Now calculate the following expected values:
 
|type="{}"}
 
|type="{}"}
$E[P_Y(X)]$ = { 0.281 3% }
+
${\rm E}\big[P_Y(X)\big] \ = \ $ { 0.281 3% }
$E[P_X(Y)]$ = { 0.281 3% }
+
${\rm E}\big[P_X(Y)\big] \ = \ $ { 0.281 3% }
  
  
{Welche der folgenden Aussagen sind zutreffend?
+
{Which of the following statements are true?
 
|type="[]"}
 
|type="[]"}
+ E[&ndash;log<sub>2</sub> <i>P<sub>U</sub></i>(<i>U</i>)] ergibt die Entropie der Zufallsgröße <i>U</i>.
+
+ ${\rm E}\big[- \log_2 \ P_U(U)\big]$&nbsp; gives the entropy of the random variable&nbsp; $U$.
+ E[&ndash;log<sub>2</sub> <i>P<sub>V</sub></i>(<i>V</i>)] ergibt die Entropie der Zufallsgröße <i>V</i>.
+
+ ${\rm E}\big[- \log_2 \ P_V(V)\big]$&nbsp; gives the entropy of the random variable&nbsp; $V$.
- E[&ndash;log<sub>2</sub> <i>P<sub>V</sub></i>(<i>U</i>)] ergibt die Entropie der Zufallsgröße <i>V</i>.
+
- ${\rm E}\big[- \log_2 \ P_V(U)\big]$&nbsp; gives the entropy of the random variable&nbsp; $V$.
  
  
Line 79: Line 79:
 
</quiz>
 
</quiz>
  
===Musterlösung===
+
===Solution===
 
{{ML-Kopf}}
 
{{ML-Kopf}}
<b>1.</b>&nbsp;&nbsp;Allgemein gilt für den Erwartungswert der Funktion <i>F</i>(<i>X</i>) hinsichtlich der Zufallsvariablen <i>X</i>:
+
'''(1)'''&nbsp; In general, the following applies to the expected value of the function&nbsp; $F(X)$&nbsp; with respect to the random variable&nbsp; $X$:
:$${\rm E} \left [ F(X)\right ] = \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} {\rm supp} (P_X)}  \hspace{0.01cm}  
+
:$${\rm E} \left [ F(X)\right ] = \hspace{-0.4cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} {\rm supp} (P_X)}  \hspace{-0.2cm}  
 
  P_{X}(x) \cdot F(x)  \hspace{0.05cm}.$$
 
  P_{X}(x) \cdot F(x)  \hspace{0.05cm}.$$
Im vorliegenden Beispiel gilt <i>X</i> = {0, 1, 2, 3} und <i>P<sub>X</sub></i>(<i>X</i>) = [1/2, 1/8, 0, 3/8]. Wegen <i>P<sub>X</sub></i>(<i>X</i> = 2) = 0 ergibt sich somit für die zu berücksichtigende Menge (&bdquo;Support&rdquo;) in obiger Summation:
+
In the present example,&nbsp; $X = \{0,\ 1,\ 2,\ 3\}$&nbsp; and&nbsp; $P_X(X) = \big [1/2, \ 1/8, \ 0, \ 3/8\big ]$.  
 +
*Because of&nbsp; $P_X(X = 2) = 0$&nbsp;, the quantity to be taken into account&nbsp; (the "support")&nbsp; in the above summation thus results in
 
:$${\rm supp} (P_X)  = \{ 0\hspace{0.05cm}, 1\hspace{0.05cm}, 3 \}  \hspace{0.05cm}.$$
 
:$${\rm supp} (P_X)  = \{ 0\hspace{0.05cm}, 1\hspace{0.05cm}, 3 \}  \hspace{0.05cm}.$$
Mit <i>F</i>(<i>X</i>) = 1/<i>P<sub>X</sub></i>(<i>X</i>) erhält man weiter:
+
*With&nbsp; $F(X) = 1/P_X(X)$&nbsp; one further obtains:
:$${\rm E} \left [ 1/P_X(X)\right ] = \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{0.01cm} P_{X}(x) \cdot {1}/{P_X(x)}  
+
:$${\rm E} \big [ 1/P_X(X)\big ] = \hspace{-0.4cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{-0.4cm} P_{X}(x) \cdot {1}/{P_X(x)}  
= \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{-0.3cm} 1  
+
= \hspace{-0.4cm}  \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{-0.3cm} 1  
 
\hspace{0.15cm}\underline{ = 3} \hspace{0.05cm}.$$
 
\hspace{0.15cm}\underline{ = 3} \hspace{0.05cm}.$$
Der zweite Erwartungswert liefert mit supp(<i>P<sub>Y</sub></i>) = {0, 1, 2} das gleiche Ergebnis: E[1/<i>P<sub>Y</sub></i>(<i>Y</i>)] <u>= 3</u>.
+
*The second expected value gives the same result with&nbsp; ${\rm supp} (P_Y) = \{ 0\hspace{0.05cm}, 1\hspace{0.05cm}, 2 \} $&nbsp;:
 +
:$${\rm E} \left [ 1/P_Y(Y)\right ] \hspace{0.15cm}\underline{ = 3}.$$
 +
 
 +
 
  
<b>2.</b>&nbsp;&nbsp;In beiden Fällen ist der Index der Wahrscheinlichkeitsfunktion mit der Zufallsvariablen (<i>X</i> bzw. <i>Y</i>) identisch und man erhält
+
'''(2)'''&nbsp; In both cases, the index of the probability mass function is identical with the random variable&nbsp; $(X$&nbsp; or &nbsp; $Y)$&nbsp; and we obtain
:$${\rm E} \left [ P_X(X)\right ] = \hspace{0.3cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{0.3cm}  P_{X}(x) \cdot {P_X(x)}  
+
:$${\rm E} \big [ P_X(X)\big ] = \hspace{-0.3cm}   \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{-0.3cm}  P_{X}(x) \cdot {P_X(x)}  
 
= (1/2)^2 + (1/8)^2 + (3/8)^2 = 13/32
 
= (1/2)^2 + (1/8)^2 + (3/8)^2 = 13/32
 
\hspace{0.15cm}\underline{ \approx 0.406} \hspace{0.05cm},$$
 
\hspace{0.15cm}\underline{ \approx 0.406} \hspace{0.05cm},$$
:$${\rm E} \left [ P_Y(Y)\right ] = \hspace{0.3cm}  \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 2 \}}  \hspace{0.3cm}  P_Y(y) \cdot P_Y(y) = (1/2)^2 + (1/4)^2 + (1/4)^2  
+
:$${\rm E} \big [ P_Y(Y)\big ] = \hspace{-0.3cm}  \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 2 \}}  \hspace{-0.3cm}  P_Y(y) \cdot P_Y(y) = (1/2)^2 + (1/4)^2 + (1/4)^2  
 
\hspace{0.15cm}\underline{ = 0.375} \hspace{0.05cm}.$$
 
\hspace{0.15cm}\underline{ = 0.375} \hspace{0.05cm}.$$
  
<b>3.</b>&nbsp;&nbsp;Hier gelten folgende Gleichungen:
 
:$${\rm E} \left [ P_Y(X)\right ] = \hspace{0.3cm}  \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 3 \}}  \hspace{0.3cm}  P_{X}(x) \cdot {P_Y(x)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{8} \cdot \frac{1}{4} + \frac{3}{8} \cdot 0 = 9/32
 
\hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm},$$
 
Die Erwartungswertbildung bezieht sich hier auf <i>P<sub>X</sub></i>(&middot;), also auf die Zufallsgröße <i>X</i>. <i>P<sub>Y</sub></i>(&middot;) ist dabei die formale Funktion ohne (direkten) Bezug zur Zufallsgröße <i>Y</i>.
 
  
Für den zweiten Erwartungswert erhält man den gleichen Zahlenwert (das muss nicht so sein):
 
:$${\rm E} \left [ P_X(Y)\right ] = \hspace{0.3cm}  \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm}, 1\hspace{0.05cm},\hspace{0.05cm} 2 \}}  \hspace{0.3cm}  P_{Y}(y) \cdot {P_X(y)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{4} \cdot \frac{1}{8} + \frac{1}{4} \cdot 0 = 9/32 \hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm}.$$
 
  
<b>4.</b>&nbsp;&nbsp;Wir berechnen zunächst die drei Erwartungswerte:
+
'''(3)'''&nbsp; The following equations apply here:
:$${\rm E} \left [-{\rm log}_2 \hspace{0.1cm} P_U(U)\right ]
+
:$${\rm E} \big [ P_Y(X)\big ] = \hspace{-0.3cm}  \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}\hspace{-0.3cm}  P_{X}(x) \cdot {P_Y(x)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{8} \cdot \frac{1}{4} + \frac{3}{8} \cdot 0 = 9/32
= \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} \hspace{0.15cm}\underline{ = 1\,{\rm bit}} \hspace{0.05cm},$$
+
\hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm},$$
:$${\rm E} \left [-{\rm log}_2 \hspace{0.1cm} P_V(V)\right ]
+
*The expected value formation here refers to&nbsp; $P_X(&middot;)$, i.e. to the random variable&nbsp; $X$.
  = \frac{3}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 0.811\,{\rm bit}} \hspace{0.05cm},$$
+
*$P_Y(&middot;)$&nbsp;is the formal function without (direct) reference to the random variable&nbsp; $Y$.
:$${\rm E} \left [-{\rm log}_2 \hspace{0.1cm} P_V(U)\right ]
+
*The same numerical value is obtained for the second expected value&nbsp; (this does not have to be the case in general):
= \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 1.208\,{\rm bit}} \hspace{0.05cm}.$$
+
:$${\rm E} \big [ P_X(Y)\big ] = \hspace{-0.3cm}  \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 2 \}} \hspace{-0.3cm}  P_{Y}(y) \cdot {P_X(y)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{4} \cdot \frac{1}{8} + \frac{1}{4} \cdot 0 = 9/32 \hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm}.$$
Richtig sind demnach die <u>beiden ersten Aussagen</u>:
 
  
:* Die Entropie <i>H</i>(<i>U</i>) = 1 bit kann entsprechend der ersten Gleichung berechnet werden. Sie gilt für die binäre Zufallsgröße <i>U</i> mit gleichen Wahrscheinlichkeiten.
 
  
:* Die Entropie <i>H</i>(<i>V</i>) = 0.811 bit berechnet sich entsprechend der zweiten Gleichung. Aufgrund der Wahrscheinlichkeiten 3/4 und 1/4 ist die Entropie (Unsicherheit) kleiner als für die Zufallsgröße <i>U</i>.
 
  
:* Der dritte Erwartungswert kann schon allein vom Ergebnis her (1.208 bit) nicht die Entropie einer binären Zufallsgröße angeben, die stets auf 1 bit begrenzt ist.
+
'''(4)'''&nbsp; We first calculate the three expected values:
 +
:$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_U(U)\big ]
 +
= \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} \hspace{0.15cm}\underline{ = 1\ {\rm bit}} \hspace{0.05cm},$$
 +
:$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_V(V)\big ]
 +
= \frac{3}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 0.811\ {\rm bit}} \hspace{0.05cm},$$
 +
:$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_V(U)\big ]
 +
= \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 1.208\ {\rm bit}} \hspace{0.05cm}.$$
 +
Accordingly, the <u>first two statements</u> are correct:
 +
* The entropy&nbsp; $H(U) = 1$&nbsp; bit&nbsp; can be calculated according to the first equation.&nbsp; It applies to the binary random variable&nbsp; $U$&nbsp; with equal probabilities.
 +
* The entropy&nbsp; $H(V) = 0.811$&nbsp; bit&nbsp; is calculated according to the second equation.&nbsp; Due to the probabilities&nbsp; $3/4$&nbsp; and&nbsp; $1/4$&nbsp;, the entropy (uncertainty) is smaller here than for the random variable&nbsp; $U$.
 +
* The third expected value cannot indicate the entropy of a binary random variable, which is always limited to&nbsp; $1$&nbsp; (bit)&nbsp;, simply because of the result &nbsp; $(1.208$&nbsp; bit$)$&nbsp;.
 
{{ML-Fuß}}
 
{{ML-Fuß}}
  
  
  
[[Category:Aufgaben zu Informationstheorie|^3.1 Vorbemerkungen zu 2D-Zufallsgrößen^]]
+
[[Category:Information Theory: Exercises|^3.1 General Information on 2D Random Variables^]]

Latest revision as of 09:11, 24 September 2021

Two-dimensional
probability mass function

We consider the following probability mass functions  $\rm (PMF)$:

$$P_X(X) = \big[1/2,\ 1/8,\ 0,\ 3/8 \big],$$
$$P_Y(Y) = \big[1/2,\ 1/4,\ 1/4,\ 0 \big],$$
$$P_U(U) = \big[1/2,\ 1/2 \big],$$
$$P_V(V) = \big[3/4,\ 1/4\big].$$

For the associated random variables, let:

$X= \{0,\ 1,\ 2,\ 3\}$,     $Y= \{0,\ 1,\ 2,\ 3\}$,    $U = \{0,\ 1\}$,     $V = \{0, 1\}$.

Often, for such discrete random variables, one must have to calculate different expected values of the form

$${\rm E} \big [ F(X)\big ] =\hspace{-0.3cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm}\hspace{-0.03cm} {\rm supp} (P_X)} \hspace{-0.1cm} P_{X}(x) \cdot F(x). $$

Here, denote:

  • $P_X(X)$  denotes the probability mass function of the discrete random variable   $X$.
  • The  "support"  of  $P_X$  includes all those realisations  $x$  of the random variable  $X$  with non-vanishing probability.
  • Formally, this can be written as
$${\rm supp} (P_X) = \{ x: \hspace{0.25cm}x \in X \hspace{0.15cm}\underline{\rm and} \hspace{0.15cm} P_X(x) \ne 0 \} \hspace{0.05cm}.$$
  • $F(X)$  is an (arbitrary) real-valued function that can be specified in the entire domain of definition of the random variable  $X$ .


In the task, the expected values for various functions  $F(X)$  are to be calculated, among others for

  1.   $F(X)= 1/P_X(X)$,
  2.   $F(X)= P_X(X)$,
  3.   $F(X)= - \log_2 \ P_X(X)$.





Hints:

  • The exercise belongs to the chapter  Some preliminary remarks on two-dimensional random variables.
  • The two one-dimensional probability mass functions  $P_X(X)$  and  $P_Y(Y)$  result from the presented 2D–PMF  $P_{XY}(X,\ Y)$,  as will be shown in  Exercise 3.2Z.
  • The binary probability mass functions  $P_U(U)$  and  $P_V(V)$  are obtained according to the modulo operations  $U = X \hspace{0.1cm}\text{mod} \hspace{0.1cm}2$  and  $V = Y \hspace{0.1cm}\text{mod} \hspace{0.1cm} 2$.


Questions

1

What are the results of the following expected values?

${\rm E}\big[1/P_X(X)\big] \ = \ $

${\rm E}\big[1/P_{\hspace{0.04cm}Y}(\hspace{0.02cm}Y\hspace{0.02cm})\big] \ = \ $

2

Give the following expected values:

${\rm E}\big[P_X(X)\big] \ = \ $

${\rm E}\big[P_Y(Y)\big] \ = \ $

3

Now calculate the following expected values:

${\rm E}\big[P_Y(X)\big] \ = \ $

${\rm E}\big[P_X(Y)\big] \ = \ $

4

Which of the following statements are true?

${\rm E}\big[- \log_2 \ P_U(U)\big]$  gives the entropy of the random variable  $U$.
${\rm E}\big[- \log_2 \ P_V(V)\big]$  gives the entropy of the random variable  $V$.
${\rm E}\big[- \log_2 \ P_V(U)\big]$  gives the entropy of the random variable  $V$.


Solution

(1)  In general, the following applies to the expected value of the function  $F(X)$  with respect to the random variable  $X$:

$${\rm E} \left [ F(X)\right ] = \hspace{-0.4cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} {\rm supp} (P_X)} \hspace{-0.2cm} P_{X}(x) \cdot F(x) \hspace{0.05cm}.$$

In the present example,  $X = \{0,\ 1,\ 2,\ 3\}$  and  $P_X(X) = \big [1/2, \ 1/8, \ 0, \ 3/8\big ]$.

  • Because of  $P_X(X = 2) = 0$ , the quantity to be taken into account  (the "support")  in the above summation thus results in
$${\rm supp} (P_X) = \{ 0\hspace{0.05cm}, 1\hspace{0.05cm}, 3 \} \hspace{0.05cm}.$$
  • With  $F(X) = 1/P_X(X)$  one further obtains:
$${\rm E} \big [ 1/P_X(X)\big ] = \hspace{-0.4cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}} \hspace{-0.4cm} P_{X}(x) \cdot {1}/{P_X(x)} = \hspace{-0.4cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}} \hspace{-0.3cm} 1 \hspace{0.15cm}\underline{ = 3} \hspace{0.05cm}.$$
  • The second expected value gives the same result with  ${\rm supp} (P_Y) = \{ 0\hspace{0.05cm}, 1\hspace{0.05cm}, 2 \} $ :
$${\rm E} \left [ 1/P_Y(Y)\right ] \hspace{0.15cm}\underline{ = 3}.$$


(2)  In both cases, the index of the probability mass function is identical with the random variable  $(X$  or   $Y)$  and we obtain

$${\rm E} \big [ P_X(X)\big ] = \hspace{-0.3cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}} \hspace{-0.3cm} P_{X}(x) \cdot {P_X(x)} = (1/2)^2 + (1/8)^2 + (3/8)^2 = 13/32 \hspace{0.15cm}\underline{ \approx 0.406} \hspace{0.05cm},$$
$${\rm E} \big [ P_Y(Y)\big ] = \hspace{-0.3cm} \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 2 \}} \hspace{-0.3cm} P_Y(y) \cdot P_Y(y) = (1/2)^2 + (1/4)^2 + (1/4)^2 \hspace{0.15cm}\underline{ = 0.375} \hspace{0.05cm}.$$


(3)  The following equations apply here:

$${\rm E} \big [ P_Y(X)\big ] = \hspace{-0.3cm} \sum_{x \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 3 \}} \hspace{-0.3cm} P_{X}(x) \cdot {P_Y(x)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{8} \cdot \frac{1}{4} + \frac{3}{8} \cdot 0 = 9/32 \hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm},$$
  • The expected value formation here refers to  $P_X(·)$, i.e. to the random variable  $X$.
  • $P_Y(·)$ is the formal function without (direct) reference to the random variable  $Y$.
  • The same numerical value is obtained for the second expected value  (this does not have to be the case in general):
$${\rm E} \big [ P_X(Y)\big ] = \hspace{-0.3cm} \sum_{y \hspace{0.05cm}\in \hspace{0.05cm} \{ 0\hspace{0.05cm},\hspace{0.05cm} 1\hspace{0.05cm},\hspace{0.05cm} 2 \}} \hspace{-0.3cm} P_{Y}(y) \cdot {P_X(y)} = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{4} \cdot \frac{1}{8} + \frac{1}{4} \cdot 0 = 9/32 \hspace{0.15cm}\underline{ \approx 0.281} \hspace{0.05cm}.$$


(4)  We first calculate the three expected values:

$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_U(U)\big ] = \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{2}{1} \hspace{0.15cm}\underline{ = 1\ {\rm bit}} \hspace{0.05cm},$$
$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_V(V)\big ] = \frac{3}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{4} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 0.811\ {\rm bit}} \hspace{0.05cm},$$
$${\rm E} \big [-{\rm log}_2 \hspace{0.1cm} P_V(U)\big ] = \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{3} + \frac{1}{2} \cdot {\rm log}_2 \hspace{0.1cm} \frac{4}{1} \hspace{0.15cm}\underline{ = 1.208\ {\rm bit}} \hspace{0.05cm}.$$

Accordingly, the first two statements are correct:

  • The entropy  $H(U) = 1$  bit  can be calculated according to the first equation.  It applies to the binary random variable  $U$  with equal probabilities.
  • The entropy  $H(V) = 0.811$  bit  is calculated according to the second equation.  Due to the probabilities  $3/4$  and  $1/4$ , the entropy (uncertainty) is smaller here than for the random variable  $U$.
  • The third expected value cannot indicate the entropy of a binary random variable, which is always limited to  $1$  (bit) , simply because of the result   $(1.208$  bit$)$ .