Aufgaben:Exercise 3.4: Entropy for Different PMF - Revision history

Noah at 08:13, 24 September 2021

2021-09-24T08:13:41Z

Guenter at 10:17, 31 August 2021

2021-08-31T10:17:55Z

Guenter at 10:07, 31 August 2021

2021-08-31T10:07:37Z

Guenter at 10:05, 31 August 2021

2021-08-31T10:05:46Z

Guenter at 10:05, 31 August 2021

2021-08-31T10:05:16Z

Noah at 20:19, 21 August 2021

2021-08-21T20:19:32Z

Guenter at 16:25, 13 August 2021

2021-08-13T16:25:16Z

Guenter at 13:32, 16 July 2021

2021-07-16T13:32:56Z

Guenter: Guenter moved page Aufgaben:Aufgabe 3.4: Entropie für verschiedene Wahrscheinlichkeiten to Aufgaben:Exercise 3.4: Entropy for Different PMF

2021-07-16T13:22:30Z

Guenter moved page Aufgabe 3.4: Entropie für verschiedene Wahrscheinlichkeiten to Exercise 3.4: Entropy for Different PMF

Javier: Text replacement - "Category:Aufgaben zu Informationstheorie" to "Category:Information Theory: Exercises"

2021-03-23T12:46:45Z

Text replacement - "Category:Aufgaben zu Informationstheorie" to "Category:Information Theory: Exercises"

← Older revision		Revision as of 08:13, 24 September 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Einige Vorbemerkungen zu zweidimensionalen Zufallsgrößen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables
	}}		}}

@@ Line 60: / Line 60: @@
 .4 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.4}
 \hspace{0.15cm} \underline {= 1.846}  \hspace{0.05cm}.$$
-Here (and in the other tasks) the pseudo-unit 'bit' is to be added in each case.
+Here (and in the other tasks) the pseudo-unit&nbsp; "bit"&nbsp; is to be added in each case.
-'''(2)'''&nbsp; The entropy&nbsp; $H_{\rm b}(X)$&nbsp; can be represented as the sum of two parts&nbsp;  $H_{\rm b1}(X)$&nbsp; and&nbsp; $H_{\rm b2}(X)$&nbsp;, with:
+'''(2)'''&nbsp; The entropy&nbsp; $H_{\rm b}(X)$&nbsp; can be represented as the sum of two parts&nbsp;  $H_{\rm b1}(X)$&nbsp; and&nbsp; $H_{\rm b2}(X)$,&nbsp; with:
 :$$H_{\rm b1}(X) =
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} +
@@ Line 82: / Line 82: @@
-'''(3)'''&nbsp; Analogous to subtask&nbsp; '''(2)'''&nbsp;,&nbsp; $p_1 = 0.1$&nbsp; and&nbsp; $p_4 = 0.4$&nbsp; yield the maximum for&nbsp; $p_2 = p_3  = 0.25$:
+'''(3)'''&nbsp; Analogous to subtask&nbsp; '''(2)''',&nbsp; $p_1 = 0.1$&nbsp; and&nbsp; $p_4 = 0.4$&nbsp; yield the maximum for&nbsp; $p_2 = p_3  = 0.25$:
 :$$H_{\rm c}(X) =
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} +

@@ Line 24: / Line 24: @@
 Hints:
-*The exercise belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on 2D random variables]].
+*The exercise belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on two-dimensional random variables]].
 *In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy|Probability mass function and entropy]].

@@ Line 4: / Line 4: @@
 [[File:EN_Inf_Z_3_3.png|right|frame|Probability functions, each with&nbsp; $M = 4$&nbsp; elements]]
-In the first row of the adjacent table, the probability function denoted by&nbsp; $\rm (a)$&nbsp; is given in the following.
+In the first row of the adjacent table, the probability mass function denoted by&nbsp; $\rm (a)$&nbsp; is given in the following.
 For this PMF&nbsp; $P_X(X) = \big [0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; the entropy is to be calculated in subtask&nbsp; '''(1)'''&nbsp;:

@@ Line 4: / Line 4: @@
 [[File:EN_Inf_Z_3_3.png|right|frame|Probability functions, each with&nbsp; $M = 4$&nbsp; elements]]
-In the first row of the adjacent table, the probability function denoted by&nbsp; $\rm (a)$&nbsp; is given in the following. For this PMF&nbsp; $P_X(X) = \big [0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; the entropy is to be calculated in subtask&nbsp; '''(1)'''&nbsp;:
+In the first row of the adjacent table, the probability function denoted by&nbsp; $\rm (a)$&nbsp; is given in the following.
 :$$H_{\rm a}(X) = {\rm E} \big [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{X}(X)}\big ]= - {\rm E} \big  [ {\rm log}_2 \hspace{0.1cm}{P_{X}(X)}\big ].$$
-Since the logarithm to the base&nbsp; $2$&nbsp; is used here, the pseudo-unit 'bit' is to be added.
+Since the logarithm to the base&nbsp; $2$&nbsp; is used here, the pseudo-unit&nbsp; "bit"&nbsp; is to be added.
-In the further tasks, some probabilities are to be varied in each case in such a way that the greatest possible entropy results in each case:
+In the further tasks, some probabilities are to be varied in each case in such a way that the greatest possible entropy results:
-* By suitably varying &nbsp;$p_3$ &nbsp;and&nbsp; $p_4$&nbsp;, one arrives at the maximum entropy&nbsp; $H_{\rm b}(X)$&nbsp; under the condition &nbsp;$p_1 = 0.1$ &nbsp;and&nbsp; $p_2 = 0.2$   &nbsp; &rArr; &nbsp; subtask&nbsp; '''(2)'''.
+* By suitably varying &nbsp;$p_3$ &nbsp;and&nbsp; $p_4$,&nbsp; one arrives at the maximum entropy&nbsp; $H_{\rm b}(X)$&nbsp; under the condition &nbsp;$p_1 = 0.1$ &nbsp;and&nbsp; $p_2 = 0.2$   &nbsp; &rArr; &nbsp; subtask&nbsp; '''(2)'''.
 * By varying &nbsp;$p_2$ &nbsp;and&nbsp; $p_3$ appropriately, one arrives at the maximum entropy&nbsp; $H_{\rm c}(X)$&nbsp; under the condition &nbsp;$p_1 = 0.1$ &nbsp;and&nbsp; $p_4 = 0.4$ &nbsp; &rArr; &nbsp; subtask&nbsp; '''(3)'''.
-* In subtask&nbsp; '''(4)'''&nbsp; all four parameters are released for variation, which are to be determined according to the maximum entropy &nbsp; &rArr; &nbsp; $H_{\rm max}(X)$&nbsp;.
+* In subtask&nbsp; '''(4)'''&nbsp; all four parameters are released for variation,&nbsp; which are to be determined according to the maximum entropy &nbsp; &rArr; &nbsp; $H_{\rm max}(X)$&nbsp;.
@@ Line 23: / Line 25: @@
 Hints:
 *The exercise belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on 2D random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy|Probability Function and Entropy]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy|Probability mass function and entropy]].
@@ Line 30: / Line 32: @@
 <quiz display=simple>
-{To which entropy does the probability function&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$ lead?
+{To which entropy does the probability mass function&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$ lead?
 |type="{}"}
 $H_{\rm a}(X) \ = \ $ { 1.846 0.5% } $\ \rm bit$

@@ Line 3: / Line 3: @@
 }}
-[[File:EN_Inf_Z_3_3.png|right|frame|Wahrscheinlichkeitsfunktionen, jeweils mit&nbsp; $M = 4$&nbsp; Elementen]]
+[[File:EN_Inf_Z_3_3.png|right|frame|Probability functions, each with&nbsp; $M = 4$&nbsp; elements]]
-In der ersten Zeile der nebenstehenden Tabelle ist die im Folgenden die mit&nbsp; $\rm (a)$&nbsp; bezeichnete Wahrscheinlichkeitsfunktion angegeben. Für diese PMF&nbsp; $P_X(X) = \big [0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; soll  in der Teilaufgabe&nbsp; '''(1)'''&nbsp; die Entropie berechnet werden:
+In the first row of the adjacent table, the probability function denoted by&nbsp; $\rm (a)$&nbsp; is given in the following. For this PMF&nbsp; $P_X(X) = \big [0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; the entropy is to be calculated in subtask&nbsp; '''(1)'''&nbsp;:
 :$$H_{\rm a}(X) = {\rm E} \big [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{X}(X)}\big ]= - {\rm E} \big  [ {\rm log}_2 \hspace{0.1cm}{P_{X}(X)}\big ].$$
-Da hier der Logarithmus zur Basis&nbsp; $2$&nbsp; verwendet wird, ist die Pseudo–Einheit „bit” anzufügen.
+Since the logarithm to the base&nbsp; $2$&nbsp; is used here, the pseudo-unit 'bit' is to be added.
-In den weiteren Aufgaben sollen jeweils einige Wahrscheinlichkeiten variiert werden und zwar derart, dass sich jeweils die größtmögliche Entropie ergibt:
+In the further tasks, some probabilities are to be varied in each case in such a way that the greatest possible entropy results in each case:
-* Durch geeignete Variation von &nbsp;$p_3$ &nbsp;und&nbsp; $p_4$&nbsp; kommt man zur maximalen Entropie&nbsp; $H_{\rm b}(X)$&nbsp; unter der Voraussetzung &nbsp;$p_1 = 0.1$ &nbsp;und&nbsp; $p_2 = 0.2$   &nbsp; &rArr; &nbsp; Teilaufgabe&nbsp; '''(2)'''.
+* By suitably varying &nbsp;$p_3$ &nbsp;and&nbsp; $p_4$&nbsp;, one arrives at the maximum entropy&nbsp; $H_{\rm b}(X)$&nbsp; under the condition &nbsp;$p_1 = 0.1$ &nbsp;and&nbsp; $p_2 = 0.2$   &nbsp; &rArr; &nbsp; subtask&nbsp; '''(2)'''.
-* Durch geeignete Variation von &nbsp;$p_2$ &nbsp;und&nbsp; $p_3$ kommt man zur maximalen Entropie&nbsp; $H_{\rm c}(X)$&nbsp; unter der Voraussetzung &nbsp;$p_1 = 0.1$ &nbsp;und&nbsp; $p_4 = 0.4$ &nbsp; &rArr; &nbsp; Teilaufgabe&nbsp; '''(3)'''.
+* By varying &nbsp;$p_2$ &nbsp;and&nbsp; $p_3$ appropriately, one arrives at the maximum entropy&nbsp; $H_{\rm c}(X)$&nbsp; under the condition &nbsp;$p_1 = 0.1$ &nbsp;and&nbsp; $p_4 = 0.4$ &nbsp; &rArr; &nbsp; subtask&nbsp; '''(3)'''.
-* In der Teilaufgabe&nbsp; '''(4)'''&nbsp; sind alle vier Parameter zur Variation freigegeben, die entsprechend der maximalen Entropie &nbsp; &rArr; &nbsp; $H_{\rm max}(X)$&nbsp;  zu bestimmen sind.
+* In subtask&nbsp; '''(4)'''&nbsp; all four parameters are released for variation, which are to be determined according to the maximum entropy &nbsp; &rArr; &nbsp; $H_{\rm max}(X)$&nbsp;.
@@ Line 21: / Line 21: @@
+Hints:
-''Hinweise:''
+*The exercise belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on 2D random variables]].
-*Die Aufgabe gehört zum  Kapitel&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Einige Vorbemerkungen zu den 2D-Zufallsgrößen]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy|Probability Function and Entropy]].
-===Fragebogen===
+===Questions===
 <quiz display=simple>
-{Zu welcher Entropie führt die Wahrscheinlichkeitsfunktion&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$ ?
+{To which entropy does the probability function&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$ lead?
 |type="{}"}
 $H_{\rm a}(X) \ = \ $ { 1.846 0.5% } $\ \rm bit$
-{Es gelte nun allgemein&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ p_3, \ p_4\big ]$.&nbsp; Welche Entropie erhält man, wenn&nbsp; $p_3$&nbsp; und&nbsp; $p_4$&nbsp; bestmöglich gewählt werden?
+{Let&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ p_3, \ p_4\big ]$ apply in general.&nbsp; What entropy is obtained if&nbsp; $p_3$&nbsp; and&nbsp; $p_4$&nbsp; are chosen as best as possible?
 |type="{}"}
 $H_{\rm b}(X) \ = \ $ { 1.857 0.5% } $\ \rm bit$
-{ Nun gelte&nbsp; $P_X(X) = \big [ 0.1, \ p_2, \ p_3, \ 0.4 \big ]$.&nbsp; Welche Entropie erhält man, wenn&nbsp; $p_2$&nbsp; und&nbsp; $p_3$&nbsp; bestmöglich gewählt werden?
+{ Now let&nbsp; $P_X(X) = \big [ 0.1, \ p_2, \ p_3, \ 0.4 \big ]$.&nbsp; What entropy is obtained if&nbsp; $p_2$&nbsp; and&nbsp; $p_3$&nbsp; are chosen as best as possible?
 |type="{}"}
 $H_{\rm c}(X) \ = \ $ { 1.861 0.5% } $\ \rm bit$
-{ Welche Entropie erhält man, wenn alle Wahrscheinlichkeiten &nbsp;$(p_1, \ p_2 , \ p_3, \ p_4)$&nbsp; bestmöglich gewählt werden können?
+{ What entropy is obtained if all probabilities &nbsp;$(p_1, \ p_2 , \ p_3, \ p_4)$&nbsp; can be chosen as best as possible?
 |type="{}"}
 $H_{\rm max}(X) \ = \ $ { 2 1% } $\ \rm bit$
@@ Line 50: / Line 49: @@
 </quiz>
-===Musterlösung===
+===Solution===
 {{ML-Kopf}}
-'''(1)'''&nbsp;  Mit&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; erhält man für die Entropie:
+'''(1)'''&nbsp;  With&nbsp; $P_X(X) = \big [ 0.1, \ 0.2, \ 0.3, \ 0.4 \big ]$&nbsp; we get for the entropy:
 :$$H_{\rm a}(X) =
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} +
@@ Line 59: / Line 58: @@
 .4 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.4}
 \hspace{0.15cm} \underline {= 1.846}  \hspace{0.05cm}.$$
-Hier (und bei den anderen Aufgaben) ist jeweils die  Pseudo–Einheit „bit” anzufügen.
+Here (and in the other tasks) the pseudo-unit 'bit' is to be added in each case.
-'''(2)'''&nbsp; Die Entropie&nbsp; $H_{\rm b}(X)$&nbsp; lässt sich als Summe zweier Anteile&nbsp;  $H_{\rm b1}(X)$&nbsp; und&nbsp; $H_{\rm b2}(X)$&nbsp;  darstellen, mit:
+'''(2)'''&nbsp; The entropy&nbsp; $H_{\rm b}(X)$&nbsp; can be represented as the sum of two parts&nbsp;  $H_{\rm b1}(X)$&nbsp; and&nbsp; $H_{\rm b2}(X)$&nbsp;, with:
 :$$H_{\rm b1}(X) =
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} +
@@ Line 71: / Line 70: @@
 (0.7-p_3) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.7-p_3} \hspace{0.05cm}.$$
-*Die zweite Funktion ist  maximal für&nbsp; $p_3 = p_4 = 0.35$.&nbsp; Ein ähnlicher Zusammenhang hat sich bei der binären Entropiefunktion ergeben.&nbsp;
+*The second function is maximum for&nbsp; $p_3 = p_4 = 0.35$.&nbsp; A similar relationship has been found for the binary entropy function. &nbsp;
-*Damit erhält man:
+*Thus one obtains:
 :$$H_{\rm b2}(X) = 2 \cdot
@@ Line 81: / Line 80: @@
-'''(3)'''&nbsp; Analog zur Teilaufgabe&nbsp; '''(2)'''&nbsp; ergibt sich mit&nbsp; $p_1 = 0.1$&nbsp; und&nbsp; $p_4 = 0.4$&nbsp; das Maximum für&nbsp; $p_2 = p_3  = 0.25$:
+'''(3)'''&nbsp; Analogous to subtask&nbsp; '''(2)'''&nbsp;,&nbsp; $p_1 = 0.1$&nbsp; and&nbsp; $p_4 = 0.4$&nbsp; yield the maximum for&nbsp; $p_2 = p_3  = 0.25$:
 :$$H_{\rm c}(X) =
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} +
@@ Line 90: / Line 89: @@
-'''(4)'''&nbsp; Die maximale Entropie für den Symbolumfang&nbsp; $M=4$&nbsp; ergibt sich bei gleichen Wahrscheinlichkeiten, also für&nbsp; $p_1 = p_2 = p_3 = p_4 = 0.25$:
+'''(4)'''&nbsp; The maximum entropy for the symbol range&nbsp; $M=4$&nbsp; is obtained for equal probabilities, i.e. for&nbsp; $p_1 = p_2 = p_3 = p_4 = 0.25$:
 :$$H_{\rm max}(X) =
 {\rm log}_2 \hspace{0.1cm} M
 \hspace{0.15cm} \underline {= 2}  \hspace{0.05cm}.$$
-*Die Differenz der Entropien entsprechend&nbsp; '''(4)'''&nbsp; und&nbsp; '''(3)'''&nbsp; ergibt&nbsp; ${\rm \Delta} H(X) = 0.139 \ \rm bit$.&nbsp;  Hierbei gilt:
+*The difference of the entropies according to&nbsp; '''(4)'''&nbsp; and&nbsp; '''(3)'''&nbsp; gives&nbsp; ${\rm \Delta} H(X) = 0.139 \ \rm bit$.&nbsp;  Here:
 :$${\rm \Delta}  H(X) = 1-
 .1 \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{0.1} -
@@ Line 101: / Line 100: @@
   \hspace{0.05cm}.$$
-*Mit der binären Entropiefunktion
+*With the binary entropy function
 :$$H_{\rm bin}(p) =
@@ Line 107: / Line 106: @@
 (1-p) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{1-p}$$
-:lässt sich hierfür auch schreiben:
+:can also be written for this:
 :$${\rm \Delta} H(X) = 0.5 \cdot \big [ 1- H_{\rm bin}(0.2) \big ] =

← Older revision		Revision as of 13:32, 16 July 2021
Line 124:		Line 124:


−	[[Category:Information Theory: Exercises\|^3.1 ~~Allgemeines zu~~ 2D~~-Zufallsgrößen~~^]]	+	[[Category:Information Theory: Exercises\|^3.1 General Information on 2D Random Variables^]]

← Older revision	Revision as of 13:22, 16 July 2021
(No difference)

← Older revision		Revision as of 12:46, 23 March 2021
Line 124:		Line 124:


−	[[Category:~~Aufgaben zu Informationstheorie~~\|^3.1 Allgemeines zu 2D-Zufallsgrößen^]]	+	[[Category:Information Theory: Exercises\|^3.1 Allgemeines zu 2D-Zufallsgrößen^]]