Aufgaben:Exercise 3.9: Conditional Mutual Information - Revision history

Noah at 08:16, 24 September 2021

2021-09-24T08:16:25Z

Javier: Text replacement - "generalis" to "generaliz"

2021-09-22T11:48:09Z

Text replacement - "generalis" to "generaliz"

Guenter at 14:31, 21 September 2021

2021-09-21T14:31:50Z

Guenter at 14:24, 21 September 2021

2021-09-21T14:24:38Z

Noah at 10:04, 14 September 2021

2021-09-14T10:04:03Z

Noah at 11:24, 13 September 2021

2021-09-13T11:24:55Z

Noah at 11:23, 13 September 2021

2021-09-13T11:23:05Z

Show changes

Guenter at 16:51, 20 July 2021

2021-07-20T16:51:29Z

Guenter: Guenter moved page Aufgaben:Aufgabe 3.9: Bedingte Transinformation to Aufgaben:Exercise 3.9: Conditional Mutual Information

2021-07-20T16:47:38Z

Guenter moved page Aufgabe 3.9: Bedingte Transinformation to Exercise 3.9: Conditional Mutual Information

Javier: Text replacement - "Category:Aufgaben zu Informationstheorie" to "Category:Information Theory: Exercises"

2021-03-23T12:45:51Z

Text replacement - "Category:Aufgaben zu Informationstheorie" to "Category:Information Theory: Exercises"

← Older revision		Revision as of 08:16, 24 September 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Verschiedene Entropien zweidimensionaler Zufallsgrößen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Different_Entropy_Measures_of_Two-Dimensional_Random_Variables
	}}		}}

@@ Line 112: / Line 112: @@
 The result&nbsp; $I(X; W|Z) > I(X; W)$&nbsp; is true for this example, but also for many other applications:
 *If I know&nbsp; $Z$, I know more about the 2D random variable&nbsp; $XW$&nbsp; than without this knowledge..
-*However, one must not generalise this result:
+*However, one must not generalize this result:
 :Sometimes&nbsp; $I(X; W) > I(X; W|Z)$, actually applies, as in&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgr%C3%B6%C3%9Fen#Conditional_mutual_information|Example 4]]&nbsp; in the theory section.
 {{ML-Fuß}}

@@ Line 66: / Line 66: @@
 ===Solution===
 {{ML-Kopf}}
-[[File:P_ID2814__Inf_A_3_8a.png|right|frame|2D probability functions for&nbsp; $Z = 1$]]
+[[File:P_ID2814__Inf_A_3_8a.png|right|frame|Two-dimensional probability mass functions for&nbsp; $Z = 1$]]
 '''(1)'''&nbsp; The upper graph is valid for&nbsp; $Z = 1$ &nbsp; &rArr; &nbsp; $W = X + Y$.&nbsp;
 *Under the conditions&nbsp; $P_X(X) = \big [1/2, \ 1/2 \big]$&nbsp; as well as&nbsp; $P_Y(Y) = \big [1/2, \ 1/2 \big]$&nbsp; the joint probabilities&nbsp; $P_{ XW|Z=1 }(X, W)$&nbsp; thus result according to the right graph (grey background).
@@ Line 81: / Line 81: @@
 *The first term summarises the two horizontally shaded fields in the graph, the second term the vertically shaded fields.
-*The latter do not contribute because of&nbsp; $\log_2 (1) = 0$&nbsp;.
+*The second term do not contribute because of&nbsp; $\log_2 (1) = 0$&nbsp;.
-[[File:P_ID2815__Inf_A_3_8b.png|right|frame|2D probability functions for&nbsp; $Z = 2$]]
+[[File:P_ID2815__Inf_A_3_8b.png|right|frame|Two-dimensional probability mass functions for&nbsp; $Z = 2$]]
-'''(2)'''&nbsp; For&nbsp; $Z = 2$&nbsp; $W = \{4,\ 6,\ 8\}$, is valid, but nothing changes with respect to the probability functions compared to subtask&nbsp; '''(1)'''&nbsp;.
+'''(2)'''&nbsp; For&nbsp; $Z = 2$,&nbsp; $W = \{4,\ 6,\ 8\}$&nbsp; is valid, but nothing changes with respect to the probability functions compared to subtask&nbsp; '''(1)'''.
 *Consequently, the same conditional mutual information is obtained:
@@ Line 92: / Line 92: @@
 \hspace{0.15cm} \underline {=0.5\,{\rm (bit)}}
 \hspace{0.05cm}.$$
-<br clear=all>
 '''(3)'''&nbsp; The equation is for&nbsp; $Z = \{1,\ 2\}$&nbsp; with&nbsp; ${\rm Pr}(Z = 1) =p$ &nbsp;and&nbsp;  ${\rm Pr}(Z = 2) =1-p$:
 :$$I(X;W \hspace{0.05cm}|\hspace{0.05cm} Z) =  p \cdot I(X;W \hspace{0.05cm}|\hspace{0.05cm} Z = 1) + (1-p) \cdot I(X;W \hspace{0.05cm}|\hspace{0.05cm} Z = 2)\hspace{0.15cm} \underline {=0.5\,{\rm (bit)}}
@@ Line 112: / Line 113: @@
 *If I know&nbsp; $Z$, I know more about the 2D random variable&nbsp; $XW$&nbsp; than without this knowledge..
 *However, one must not generalise this result:
-:Sometimes&nbsp; $I(X; W) > I(X; W|Z)$, actually applies, as in&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgr%C3%B6%C3%9Fen#Conditional_mutual_information|example 3]] in the theory section.
+:Sometimes&nbsp; $I(X; W) > I(X; W|Z)$, actually applies, as in&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgr%C3%B6%C3%9Fen#Conditional_mutual_information|Example 4]]&nbsp; in the theory section.
 {{ML-Fuß}}

@@ Line 15: / Line 15: @@
 Three different definitions of mutual information are used in this exercise:
-*the ''conventional''&nbsp; mutual information zwischen&nbsp; $X$&nbsp; and&nbsp; $W$:
+*the&nbsp; <u>conventional</u>&nbsp; mutual information zwischen&nbsp; $X$&nbsp; and&nbsp; $W$:
 :$$I(X;W) =  H(X) - H(X|\hspace{0.05cm}W) \hspace{0.05cm},$$
-* the ''conditional''&nbsp; mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$&nbsp; with a ''given fixed value''&nbsp; $Z = z$:
+* the&nbsp; <u>conditional</u>&nbsp; mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$&nbsp; with a&nbsp; <u>given fixed value</u>&nbsp; $Z = z$:
 :$$I(X;W \hspace{0.05cm}|\hspace{0.05cm} Z = z) =  H(X\hspace{0.05cm}|\hspace{0.05cm} Z = z) - H(X|\hspace{0.05cm}W ,\hspace{0.05cm} Z = z) \hspace{0.05cm},$$
-* the ''conditional''&nbsp; mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$&nbsp; for a ''given random value''&nbsp; $Z$:
+* the&nbsp; <u>conditional</u>&nbsp; mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$&nbsp; for a&nbsp; <u>given random variable</u>&nbsp; $Z$:
 :$$I(X;W \hspace{0.05cm}|\hspace{0.05cm} Z ) =  H(X\hspace{0.05cm}|\hspace{0.05cm} Z ) - H(X|\hspace{0.05cm}W \hspace{0.05cm} Z ) \hspace{0.05cm}.$$
@@ Line 33: / Line 33: @@
 Hints:
-*How large is the mutual information between&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
+*The exercise belongs to the chapter&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
 *In particular, reference is made to the page&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Conditional_mutual_information|Conditional mutual information]].

@@ Line 11: / Line 11: @@
 From&nbsp; $X$,&nbsp; $Y$&nbsp; and&nbsp; $Z$&nbsp; we form the new random variable&nbsp; $W = (X+Y) \cdot Z$.
 * It is obvious that there are statistical dependencies between&nbsp; $X$&nbsp; and&nbsp; $W$&nbsp; &nbsp; &rArr; &nbsp; mutual information&nbsp; $I(X; W) ≠ 0$.
-*Furthermore,&nbsp; $I(Y; W) ≠ 0$ &nbsp;as well as&nbsp; $I(Z; W) ≠ 0$&nbsp; will also apply, but this will not be discussed in detail in this task.
+*Furthermore,&nbsp; $I(Y; W) ≠ 0$ &nbsp;as well as&nbsp; $I(Z; W) ≠ 0$&nbsp; will also apply, but this will not be discussed in detail in this exercise.
-Three different definitions of mutual information are used in this task:
+Three different definitions of mutual information are used in this exercise:
 *the ''conventional''&nbsp; mutual information zwischen&nbsp; $X$&nbsp; and&nbsp; $W$:
 :$$I(X;W) =  H(X) - H(X|\hspace{0.05cm}W) \hspace{0.05cm},$$

@@ Line 34: / Line 34: @@
 Hints:
 *How large is the mutual information between&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|Different entropies of two-dimensional random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Bedingte_Transinformation|Conditional mutual information]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen#Conditional_mutual_information|Conditional mutual information]].
@@ Line 40: / Line 40: @@
 <quiz display=simple>
-{How large is the transinformation between&nbsp; $X$&nbsp; and&nbsp; $W$,&nbsp; if&nbsp; $Z = 1$&nbsp; always holds?
+{How large is the mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$,&nbsp; if&nbsp; $Z = 1$&nbsp; always holds?
 |type="{}"}
 $ I(X; W | Z = 1) \ = \ $ { 0.5 3% } $\ \rm bit$
-{How large is the transinformation between&nbsp; $X$&nbsp; and&nbsp; $W$,&nbsp; if&nbsp; $Z = 2$&nbsp; always holds?
+{How large is the mutual information between&nbsp; $X$&nbsp; and&nbsp; $W$,&nbsp; if&nbsp; $Z = 2$&nbsp; always holds?
 |type="{}"}
 $ I(X; W | Z = 2) \ = \ $ { 0.5 3% } $\ \rm bit$
@@ Line 112: / Line 112: @@
 *If I know&nbsp; $Z$, I know more about the 2D random variable&nbsp; $XW$&nbsp; than without this knowledge..
 *However, one must not generalise this result:
-:Sometimes&nbsp; $I(X; W) > I(X; W|Z)$, actually applies, as in&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgr%C3%B6%C3%9Fen#Bedingte_Transinformation|example 3]] in the theory section.
+:Sometimes&nbsp; $I(X; W) > I(X; W|Z)$, actually applies, as in&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgr%C3%B6%C3%9Fen#Conditional_mutual_information|example 3]] in the theory section.
 {{ML-Fuß}}

← Older revision		Revision as of 16:51, 20 July 2021
Line 120:		Line 120:


−	[[Category:Information Theory: Exercises\|^3.2 ~~Entropien von~~ 2D~~-Zufallsgrößen~~^]]	+	[[Category:Information Theory: Exercises\|^3.2 Entropies of 2D Random Variables^]]

← Older revision	Revision as of 16:47, 20 July 2021
(No difference)

← Older revision		Revision as of 12:45, 23 March 2021
Line 120:		Line 120:


−	[[Category:~~Aufgaben zu Informationstheorie~~\|^3.2 Entropien von 2D-Zufallsgrößen^]]	+	[[Category:Information Theory: Exercises\|^3.2 Entropien von 2D-Zufallsgrößen^]]