Aufgaben:Exercise 3.8: Once more Mutual Information - Revision history

Hwang at 12:50, 17 November 2022

2022-11-17T12:50:02Z

Noah at 08:16, 24 September 2021

2021-09-24T08:16:07Z

Javier: Text replacement - "analyse" to "analyze"

2021-09-22T11:47:48Z

Text replacement - "analyse" to "analyze"

Guenter at 16:28, 20 September 2021

2021-09-20T16:28:50Z

Guenter at 16:07, 20 September 2021

2021-09-20T16:07:52Z

Guenter at 16:06, 20 September 2021

2021-09-20T16:06:52Z

Guenter at 13:09, 20 September 2021

2021-09-20T13:09:43Z

Noah at 10:09, 14 September 2021

2021-09-14T10:09:18Z

Noah at 16:32, 10 September 2021

2021-09-10T16:32:08Z

Show changes

Guenter: Guenter moved page Aufgaben:Exercise 3.8: Once more Muual Information to Aufgaben:Exercise 3.8: Once more Mutual Information

2021-07-21T12:01:12Z

Guenter moved page Exercise 3.8: Once more Muual Information to Exercise 3.8: Once more Mutual Information

@@ Line 28: / Line 28: @@
 Hints:
 *The exercise belongs to the chapter&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
-*In particular, reference is made to the pages <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Conditional_probability_and_conditional_entropy|Conditional probability and conditional entropy]] as well as <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Mutual_information_between_two_random_variables|Mutual information between two random variables]].
+*In particular, reference is made to the sections <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Conditional_probability_and_conditional_entropy|Conditional probability and conditional entropy]] as well as <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Mutual_information_between_two_random_variables|Mutual information between two random variables]].
@@ Line 68: / Line 68: @@
-'''(2)'''&nbsp; From the two-dimensional probability mass function&nbsp; $P_{ XW }(X, W)$&nbsp; on the specification page, one obtains for
+'''(2)'''&nbsp; From the two-dimensional probability mass function&nbsp; $P_{ XW }(X, W)$&nbsp; in the specification section, one obtains for
 *the joint entropy:
 :$$H(XW) =  {\rm log}_2 \hspace{0.1cm} (9)
@@ Line 89: / Line 89: @@
 [[File:P_ID2770__Inf_A_3_7c.png|right|frame|Joint probability between&nbsp; $Z$&nbsp; and&nbsp; $W$]]
 '''(3)'''&nbsp;  The second graph shows the joint probability&nbsp; $P_{ ZW }(⋅)$.&nbsp;
-*The scheme consists of&nbsp; $5 · 9 = 45$&nbsp; fields in contrast to the plot of&nbsp; $P_{ XW }(⋅)$&nbsp; on the data page with&nbsp; $3 · 9 = 27$&nbsp; fields.
+*The scheme consists of&nbsp; $5 · 9 = 45$&nbsp; fields in contrast to the plot of&nbsp; $P_{ XW }(⋅)$&nbsp; in the data section with&nbsp; $3 · 9 = 27$&nbsp; fields.
 *However, of the&nbsp; $45$&nbsp; fields, only nine are also assigned non-zero probabilities.&nbsp; The following applies to the joint entropy:  &nbsp; $H(ZW)  = 3.170\ {\rm (bit)} \hspace{0.05cm}.$
 *With the further entropies&nbsp; $H(Z)  = 3.170\ {\rm (bit)}\hspace{0.05cm}$&nbsp; and&nbsp; $H(W)  = 2.197\ {\rm (bit)}\hspace{0.05cm}$&nbsp; according to&nbsp; [[Aufgaben:Exercise_3.8Z:_Tuples_from_Ternary_Random_Variables| Exercise 3.8Z]]&nbsp; or the subquestion&nbsp; '''(2)'''&nbsp; of this exercise, one obtains for the mutual information:

← Older revision		Revision as of 08:16, 24 September 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Verschiedene Entropien zweidimensionaler Zufallsgrößen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Different_Entropy_Measures_of_Two-Dimensional_Random_Variables
	}}		}}

@@ Line 9: / Line 9: @@
 The joint probability function&nbsp; $P_{ XY }(X, Y)$&nbsp; of both random variables is given in the upper graph.&nbsp;
-In&nbsp; [[Aufgaben:Exercise_3.8Z:_Tuples_from_Ternary_Random_Variables|Exercise 3.8Z]]&nbsp; this constellation is analysed in detail.&nbsp; One obtains as a result (all data in "bit"):
+In&nbsp; [[Aufgaben:Exercise_3.8Z:_Tuples_from_Ternary_Random_Variables|Exercise 3.8Z]]&nbsp; this constellation is analyzed in detail.&nbsp; One obtains as a result (all data in "bit"):
 :* $H(X) = H(Y) = \log_2 (3) = 1.585,$
 :* $H(XY) = \log_2 (9) = 3.170,$

@@ Line 59: / Line 59: @@
 ===Solution===
 {{ML-Kopf}}
-'''(1)'''&nbsp; The<u>correct solutions are 1 and 2</u>:
+'''(1)'''&nbsp; The <u>correct solutions are 1 and 2</u>:
 *With&nbsp; $X = \{0,\ 1,\ 2\}$,&nbsp; $Y = \{0,\ 1,\ 2\}$&nbsp;, &nbsp; $X + Y = \{0,\ 1,\ 2,\ 3,\ 4\}$ holds.&nbsp;
 *The probabilities also agree with the given probability function.
-*Checking the other two specifications shows that $W = X – Y + 2$ is also possible, but not $W = Y – X + 2$.
+*Checking the other two specifications shows that&nbsp; $W = X - Y + 2$&nbsp; is also possible, but not&nbsp; $W = Y - X + 2$.
-'''(2)'''&nbsp; From the 2D probability function&nbsp; $P_{ XW }(X, W)$&nbsp; on the specification page, one obtains for
+'''(2)'''&nbsp; From the two-dimensional probability mass function&nbsp; $P_{ XW }(X, W)$&nbsp; on the specification page, one obtains for
 *the joint entropy:
 :$$H(XW) =  {\rm log}_2 \hspace{0.1cm} (9)
@@ Line 73: / Line 75: @@
 * the probability function of the random variable&nbsp; $W$:
 :$$P_W(W) = \big [\hspace{0.05cm}1/9\hspace{0.05cm}, \hspace{0.15cm} 2/9\hspace{0.05cm},\hspace{0.15cm} 3/9 \hspace{0.05cm}, \hspace{0.15cm} 2/9\hspace{0.05cm}, \hspace{0.15cm} 1/9\hspace{0.05cm} \big ]\hspace{0.05cm},$$
-*the entropy of the random variable $W$:
+*the entropy of the random variable&nbsp; $W$:
 :$$H(W) = 2 \cdot \frac{1}{9} \cdot {\rm log}_2 \hspace{0.1cm} \frac{9}{1} + 2 \cdot \frac{2}{9} \cdot {\rm log}_2 \hspace{0.1cm} \frac{9}{2} +
 \frac{3}{9} \cdot {\rm log}_2 \hspace{0.1cm} \frac{9}{3}
   {= 2.197\ {\rm (bit)}} \hspace{0.05cm}.$$
-Thus, with&nbsp; $H(X) = 1.585 \ \rm bit$&nbsp; (was given), the result for the&nbsp; ''Mutual Information'':
+Thus, with&nbsp; $H(X) = 1.585 \ \rm bit$&nbsp; (was given), the result for the&nbsp; mutual information:
 :$$I(X;W) = H(X) + H(W) - H(XW) = 1.585 + 2.197- 3.170\hspace{0.15cm} \underline {= 0.612\ {\rm (bit)}} \hspace{0.05cm}.$$
 The left of the two diagrams illustrates the calculation of the mutual information&nbsp; $I(X; W)$&nbsp; between the first component&nbsp; $X$&nbsp; and the sum&nbsp; $W$.
-We attempt an interpretation of these numerical results:
+[[File:P_ID2770__Inf_A_3_7c.png|right|frame|Joint probability between&nbsp; $Z$&nbsp; and&nbsp; $W$]]
-* The joint probability&nbsp; $P_{ ZW }(⋅)$&nbsp;, like&nbsp; $P_{ XW }(⋅)$&nbsp;, is composed of nine equally probable elements unequal to 0. It is thus obvious that the compound entropies are also equal &nbsp; ⇒ &nbsp; $H(ZW) =  H(XW) = 3.170 \ \rm (bit)$.
+'''(3)'''&nbsp;  The second graph shows the joint probability&nbsp; $P_{ ZW }(⋅)$.&nbsp;
-* If I know the tuple&nbsp; $Z = (X, Y)$&nbsp;, I naturally also know the sum&nbsp; $W = X + Y$.&nbsp; Thus&nbsp; $H(W|Z) = 0$.
+*The scheme consists of&nbsp; $5 · 9 = 45$&nbsp; fields in contrast to the plot of&nbsp; $P_{ XW }(⋅)$&nbsp; on the data page with&nbsp; $3 · 9 = 27$&nbsp; fields.
 *In contrast,&nbsp; $H(Z|W) \ne 0$.&nbsp; Rather,&nbsp; $H(Z|W) = H(X|W) = 0.973   \ \rm (bit)$.
 *The random variable&nbsp; $W$&nbsp; thus provides exactly the same information with regard to the tuple&nbsp; $Z$&nbsp; as for the individual component&nbsp; $X$.&nbsp; This is the verbal interpretation of the statement&nbsp; $H(Z|W) = H(X|W)$.
-*The joint information of&nbsp; $Z$&nbsp; and&nbsp; $W$&nbsp; &nbsp; ⇒ &nbsp; $I(Z; W)$&nbsp; is greater than the joint information of&nbsp; $X$&nbsp; and&nbsp; $W$  &nbsp; ⇒ &nbsp; $I(X; W)$, because&nbsp; $H(W|Z) =0$&nbsp;, while&nbsp; $H(W|X)$&nbsp; is non-zero, namely exactly as great as&nbsp; $H(X)$ :
+*The joint information of&nbsp; $Z$&nbsp; and&nbsp; $W$&nbsp; &nbsp; ⇒ &nbsp; $I(Z; W)$&nbsp; is greater than the joint information of&nbsp; $X$&nbsp; and&nbsp; $W$  &nbsp; ⇒ &nbsp; $I(X; W)$,&nbsp; because&nbsp; $H(W|Z) =0$,&nbsp; while&nbsp; $H(W|X)$&nbsp; is non-zero, namely exactly as great as&nbsp; $H(X)$ :
 :$$I(Z;W)  = H(W) - H(W|Z) = 2.197 - 0= 2.197\,{\rm (bit)} \hspace{0.05cm},$$
 :$$I(X;W) =  H(W) - H(W|X) = 2.197 - 1.585= 0.612\,{\rm (bit)} \hspace{0.05cm}.$$

@@ Line 9: / Line 9: @@
 The joint probability function&nbsp; $P_{ XY }(X, Y)$&nbsp; of both random variables is given in the upper graph.&nbsp;
-In&nbsp; [[Aufgaben:3.07Z_Tupel_aus_tern%C3%A4ren_Zufallsgr%C3%B6%C3%9Fen|Exercise 3.8Z]]&nbsp; this constellation is analysed in detail.&nbsp; One obtains as a result (all data in "bit"):
+In&nbsp; [[Aufgaben:Exercise_3.8Z:_Tuples_from_Ternary_Random_Variables|Exercise 3.8Z]]&nbsp; this constellation is analysed in detail.&nbsp; One obtains as a result (all data in "bit"):
 :* $H(X) = H(Y) = \log_2 (3) = 1.585,$
 :* $H(XY) = \log_2 (9) = 3.170,$

@@ Line 3: / Line 3: @@
 }}
-[[File:P_ID2768__Inf_A_3_7_neu.png|right|frame|PMF&nbsp; $P_{ XY }$,&nbsp; $P_{ XW }$]]
+[[File:P_ID2768__Inf_A_3_7_neu.png|right|frame|2D&ndash;Functions&nbsp; <br>&nbsp;$P_{ XY }$&nbsp; und&nbsp; $P_{ XW }$]]
 We consider the tuple&nbsp; $Z = (X, Y)$, where the individual components&nbsp; $X$&nbsp; and&nbsp; $Y$&nbsp; each represent ternary random variables:
 :$$X = \{ 0 ,\ 1 ,\ 2 \} , \hspace{0.3cm}Y= \{ 0 ,\ 1 ,\ 2 \}.$$
@@ Line 51: / Line 51: @@
 {Which of the following statements are true?
 |type="[]"}
-+ &nbsp; $H(ZW) = H(XW)$ is true.
++ &nbsp; $H(ZW) = H(XW)$&nbsp; is true.
-+ &nbsp; $H(W|Z) = 0$ is true.
++ &nbsp; $H(W|Z) = 0$&nbsp; is true.
-+ &nbsp; $I(Z; W) > I(X; W)$ is true.
++ &nbsp; $I(Z; W) > I(X; W)$&nbsp; is true.
 </quiz>

@@ Line 9: / Line 9: @@
 The joint probability function&nbsp; $P_{ XY }(X, Y)$&nbsp; of both random variables is given in the upper graph.&nbsp;
-In&nbsp; [[Aufgaben:3.07Z_Tupel_aus_tern%C3%A4ren_Zufallsgr%C3%B6%C3%9Fen|task 3.8Z]]&nbsp; this constellation is analysed in detail.&nbsp; One obtains as a result (all data in "bit"):
+In&nbsp; [[Aufgaben:3.07Z_Tupel_aus_tern%C3%A4ren_Zufallsgr%C3%B6%C3%9Fen|Exercise 3.8Z]]&nbsp; this constellation is analysed in detail.&nbsp; One obtains as a result (all data in "bit"):
 :* $H(X) = H(Y) = \log_2 (3) = 1.585,$
 :* $H(XY) = \log_2 (9) = 3.170,$
@@ Line 18: / Line 18: @@
 Furthermore, we consider the random variable&nbsp; $W = \{ 0,\ 1,\ 2,\ 3,\ 4 \}$, whose properties result from the composite probability function&nbsp; $P_{ XW }(X, W)$&nbsp; according to the sketch below.&nbsp; The probabilities are zero in all fields with a white background.
-What is sought in the present task is the transinformation between
+What is sought in the present exercise is the mutual information between
 :*the random variables&nbsp; $X$&nbsp; and&nbsp; $W$ &nbsp; ⇒ &nbsp;  $I(X; W)$,
 :*the random variables&nbsp; $Z$&nbsp; and&nbsp; $W &nbsp; ⇒ &nbsp; I(Z; W)$.
@@ Line 27: / Line 27: @@
 Hints:
-*The task belongs to the chapter&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
+*The exercise belongs to the chapter&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
 *In particular, reference is made to the pages <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Conditional_probability_and_conditional_entropy|Conditional probability and conditional entropy]] as well as <br> &nbsp; &nbsp;  [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Mutual_information_between_two_random_variables|Mutual information between two random variables]].
@@ Line 87: / Line 87: @@
 '''(3)'''&nbsp;  The second graph shows the joint probability&nbsp; $P_{ ZW }(⋅)$.&nbsp; The scheme consists of&nbsp; $5 · 9 = 45$&nbsp; fields in contrast to the plot of&nbsp; $P_{ XW }(⋅)$&nbsp; on the data page with&nbsp; $3 · 9 = 27$&nbsp; fields.
 *However, of the&nbsp; $45$&nbsp; fields, only nine are also assigned non-zero probabilities.&nbsp; The following applies to the compound entropy:  &nbsp; $H(ZW)  = 3.170\ {\rm (bit)} \hspace{0.05cm}.$
-*With the further entropies&nbsp; $H(Z)  = 3.170\ {\rm (bit)}\hspace{0.05cm}$&nbsp; and&nbsp; $H(W)  = 2.197\ {\rm (bit)}\hspace{0.05cm}$&nbsp; according&nbsp; [[Aufgaben:3.07Z_Tupel_aus_tern%C3%A4ren_Zufallsgr%C3%B6%C3%9Fen| task 3.8Z]]&nbsp; or the subquestion&nbsp; '''(2)'''&nbsp; of this task, one obtains for the mutual information:
+*With the further entropies&nbsp; $H(Z)  = 3.170\ {\rm (bit)}\hspace{0.05cm}$&nbsp; and&nbsp; $H(W)  = 2.197\ {\rm (bit)}\hspace{0.05cm}$&nbsp; according&nbsp; [[Aufgaben:3.07Z_Tupel_aus_tern%C3%A4ren_Zufallsgr%C3%B6%C3%9Fen| Exercise 3.8Z]]&nbsp; or the subquestion&nbsp; '''(2)'''&nbsp; of this exercise, one obtains for the mutual information:
 :$$I(Z;W) = H(Z) + H(W) - H(ZW) \hspace{0.15cm} \underline {= 2.197\,{\rm (bit)}} \hspace{0.05cm}.$$

← Older revision	Revision as of 12:01, 21 July 2021
(No difference)