Aufgaben:Exercise 1.1Z: Binary Entropy Function - Revision history

Javier: Text replacement - "optimisation" to "optimization"

2021-09-21T12:50:19Z

Text replacement - "optimisation" to "optimization"

Guenter at 12:00, 10 August 2021

2021-08-10T12:00:04Z

Guenter at 14:46, 16 June 2021

2021-06-16T14:46:36Z

Guenter at 14:39, 16 June 2021

2021-06-16T14:39:24Z

Javier: Text replacement - "”" to """

2021-05-28T14:33:03Z

Text replacement - "”" to """

Javier: Text replacement - "„" to """

2021-05-28T14:32:53Z

Text replacement - "„" to """

Guenter at 12:34, 13 May 2021

2021-05-13T12:34:17Z

Noah at 19:56, 6 May 2021

2021-05-06T19:56:36Z

Noah at 20:53, 5 May 2021

2021-05-05T20:53:18Z

Noah at 19:38, 5 May 2021

2021-05-05T19:38:22Z

@@ Line 77: / Line 77: @@
-'''(2)'''&nbsp; The optimisation condition is&nbsp; ${\rm d}H_{\rm bin}(p)/{\rm d}p = 0$&nbsp; resp.
+'''(2)'''&nbsp; The optimization condition is&nbsp; ${\rm d}H_{\rm bin}(p)/{\rm d}p = 0$&nbsp; resp.
 :$$\frac{{\rm d}H'_{\rm bin}(p)}{{\rm d}p} \stackrel{!}{=} 0 \hspace{0.3cm}\Rightarrow \hspace{0.3cm} \frac{\rm d}{{\rm d}p}
     \big [ - p \cdot {\rm ln}\hspace{0.1cm}p - (1-p) \cdot {\rm ln}\hspace{0.1cm}({1-p})\big ] \stackrel{!}{=} 0$$

← Older revision		Revision as of 12:00, 10 August 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Gedächtnislose Nachrichtenquellen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Discrete_Memoryless_Sources
	}}		}}

@@ Line 84: / Line 84: @@
 :$$\Rightarrow \hspace{0.3cm} {\rm ln}\hspace{0.1cm}\frac {1-p}{p}= 0 \hspace{0.3cm}\Rightarrow \hspace{0.3cm}\frac {1-p}{p}= 1
   \hspace{0.3cm}\Rightarrow \hspace{0.3cm} \underline { p = 0.5}\hspace{0.05cm}.$$
-*Die Entropiewerte für&nbsp; $p = 0.5$&nbsp; lauten somit:
+*The entropy values for&nbsp; $p = 0.5$&nbsp; are thus:
 :$$H'_{\rm bin}(p = 0.5) \hspace{0.1cm}  =  \hspace{0.1cm}  -2 \cdot 0.5 \cdot {\rm ln}\hspace{0.1cm}0.5 = {\rm ln}\hspace{0.1cm}2 = 0.693 \, {\rm nat}\hspace{0.05cm},$$
 :$$  H_{\rm bin}(p = 0.5) \hspace{0.1cm}  =  \hspace{0.1cm}  -2 \cdot 0.5 \cdot {\rm ld}\hspace{0.1cm}0.5 = {\rm log_2}\hspace{0.1cm}2 \hspace{0.15cm}\underline {= 1 \, {\rm bit}}\hspace{0.05cm}.$$
@@ Line 101: / Line 101: @@
   H_{\rm bin}(p = 0.11) \approx 0.5 \, {\rm bit} \hspace{0.3cm}
 \Rightarrow \hspace{0.3cm}p_1 \approx 0.11\hspace{0.05cm}. $$
-*The second solution results from the symmetry of &nbsp;  $H_{\rm bin}(p)$&nbsp; zu &nbsp;$p_2 = 1 -p_1 \hspace{0.15cm}\underline{= 0.89}$.
+*The second solution results from the symmetry of &nbsp;  $H_{\rm bin}(p)$&nbsp; to &nbsp;$p_2 = 1 -p_1 \hspace{0.15cm}\underline{= 0.89}$.
@@ Line 115: / Line 115: @@
-'''(6)'''&nbsp; &nbsp; $H_{\rm bin}(p = 0.045) = 0.265\hspace{0.05cm}\rm  bit$ is valid..
+'''(6)'''&nbsp; &nbsp; $H_{\rm bin}(p = 0.045) = 0.265\hspace{0.05cm}\rm  bit$ is valid.
 *With the result of subtask&nbsp; '''(3)''' &nbsp; &#8658; &nbsp; $H_{\rm bin}(p = 0.05) = 0.286\hspace{0.05cm}\rm  bit$&nbsp; it follows for the relative error with respect to the entropy:
 :$$\varepsilon_{H} = \frac{H_{\rm bin}(p = 0.045)- H_{\rm bin}(p= 0.05)}{H_{\rm bin}(p = 0.05)}= \frac{0.265- 0.286}{0.286}\hspace{0.15cm}\underline {= -7.3 \, {\rm \%}}
 \hspace{0.05cm}.$$
 *The result shows:
-#&nbsp;An incorrect determination of the symbol probabilities by&nbsp; $10\%$&nbsp; is much more noticeable for&nbsp; $p = 0.05$&nbsp; due to the steeper&nbsp; $H_{\rm bin}(p)$&ndash;Verlaufs deutlich stärker bemerkbar als für&nbsp; $p = 0.5$.
+#&nbsp;An incorrect determination of the symbol probabilities by&nbsp; $10\%$&nbsp; is much more noticeable for&nbsp; $p = 0.05$&nbsp; due to the steeper&nbsp; $H_{\rm bin}(p)$ course than for&nbsp; $p = 0.5$.
-#&nbsp;A too large probability&nbsp; $p = 0.055$&nbsp; would have led to&nbsp; $H_{\rm bin}(p = 0.055) = 0.307\hspace{0.05cm}\rm  bit$&nbsp; and thus to a distortion of&nbsp; $\varepsilon_H = +7.3\%$.&nbsp; In this range, the entropy curve is thus linear (with a good approximation).
+#&nbsp;A too large probability&nbsp; $p = 0.055$&nbsp; would have led to&nbsp; $H_{\rm bin}(p = 0.055) = 0.307\hspace{0.05cm}\rm  bit$&nbsp; and thus to a distortion of&nbsp; $\varepsilon_H = +7.3\%$.&nbsp;
 {{ML-Fuß}}

@@ Line 11: / Line 11: @@
 In these equations, the shorthand terms used are:
+* the&nbsp; "natural"&nbsp; logarithm &nbsp; &rArr; &nbsp; $ {\ln} \hspace{0.09cm} p = \log_{\rm e} \hspace{0.05cm} p$,
-* the <i>natural</i>&nbsp; logarithm &nbsp; &rArr; &nbsp; $ {\ln} \hspace{0.09cm} p = \log_{\rm e} \hspace{0.05cm} p$,
+* the&nbsp; "binary"&nbsp; logarithm &nbsp; &rArr; &nbsp; ${\rm ld} \hspace{0.09cm} p = \log_2 \hspace{0.05cm} p$.
 The plot shows the binary entropy function as a function of the parameter&nbsp; $p$, assuming&nbsp; $0 ≤ p ≤ 1$&nbsp;.
-In subtasks&nbsp; '''(5)'''&nbsp; and&nbsp; '''(6)'''&nbsp; the relative error is to be determined if the symbol probability &nbsp;$p$&nbsp; was determined by simulation&nbsp; $($i.e., as a relative frequency&nbsp; $h)$&nbsp;, resulting in &nbsp;$h = 0.9 \cdot p$&nbsp; by mistake.&nbsp; The relative error is then given as follows:
+In subtasks&nbsp; '''(5)'''&nbsp; and&nbsp; '''(6)'''&nbsp; the relative error is to be determined if the symbol probability &nbsp;$p$&nbsp; was determined by simulation&nbsp; $($i.e., as a relative frequency&nbsp; $h)$,&nbsp; resulting in &nbsp;$h = 0.9 \cdot p$&nbsp; by mistake.&nbsp; The relative error is then given as follows:
 :$$\varepsilon_{H} = \frac{H_{\rm bin}(h)- H_{\rm bin}(p)}{H_{\rm bin}(p)}\hspace{0.05cm}.$$
@@ Line 56: / Line 55: @@
-{Due to insufficient simulation,&nbsp; $p = 0.5$&nbsp; was determined&nbsp; $10\%$ too low.&nbsp; What is the percentage error with respect to entropy?
+{Due to insufficient simulation,&nbsp; $p = 0.5$&nbsp; was determined&nbsp; $10\%$ too low.&nbsp; What is the percentage error with respect to the entropy?
 |type="{}"}
 $p = 0.45\ \ {\rm instead of}\ \ p=0.5\hspace{-0.1cm}:\ \ \varepsilon_H \ =  \ $ { -0.72--0.68 } $\ \rm \%$
-{Due to insufficient simulation,&nbsp; $p = 0.05$&nbsp; was determined&nbsp; $10\%$&nbsp; too low.&nbsp; What is the percentage error with respect to entropy here?
+{Due to insufficient simulation,&nbsp; $p = 0.05$&nbsp; was determined&nbsp; $10\%$&nbsp; too low.&nbsp; What is the percentage error with respect to the entropy here?
 |type="{}"}
 $p = 0.045\ \ {\rm statt}\ \ p=0.05\hspace{-0.1cm}:\ \ \varepsilon_H \ =  \ $ { -7.44--7.16 } $\ \rm \%$

@@ Line 3: / Line 3: @@
 }}
-[[File:P_ID2234__Inf_Z_1_1.png|right|frame|Binary entropy function <br>in "bits&rdquo; und "nats&rdquo;]]
+[[File:P_ID2234__Inf_Z_1_1.png|right|frame|Binary entropy function <br>in "bits" und "nats"]]
 We consider a sequence of binary random variables with the symbol set&nbsp; $\{ \rm A, \ B \}$ &nbsp; &rArr; &nbsp; $M = 2$.&nbsp; Let the probabilities of occurrence of the two symbols be&nbsp; $p_{\rm A }= p$ &nbsp;and&nbsp; $p_{\rm B } = 1 - p$.
@@ Line 34: / Line 34: @@
 <quiz display=simple>
-{How are&nbsp; $H_{\rm bin}(p)$&nbsp; related to the unit "bit&rdquo; and&nbsp; $H'_{\rm bin}(p)$&nbsp; related to the unit "nat&rdquo;?
+{How are&nbsp; $H_{\rm bin}(p)$&nbsp; related to the unit "bit" and&nbsp; $H'_{\rm bin}(p)$&nbsp; related to the unit "nat"?
 |type="[]"}
 + $H_{\rm bin}(p)$&nbsp; and&nbsp; $H'_{\rm bin}(p)$&nbsp; differ by a factor.
@@ Line 97: / Line 97: @@
-'''(4)'''&nbsp; This sub-task cannot be solved in closed form, but by "trial and error&rdquo;.
+'''(4)'''&nbsp; This sub-task cannot be solved in closed form, but by "trial and error".
 *A solution gives the result:
 :$$H_{\rm bin}(p = 0.10) = 0.469 \, {\rm bit}\hspace{0.05cm},\hspace{0.2cm}H_{\rm bin}(p = 0.12) = 0.529 \, {\rm bit}\hspace{0.05cm},\hspace{0.2cm}

← Older revision		Revision as of 12:34, 13 May 2021
Line 127:		Line 127:


−	[[Category:Information Theory: Exercises\|^1.1 ~~Gedächtnislose Nachrichtenquellen~~^]]	+	[[Category:Information Theory: Exercises\|^1.1 Memoryless Sources^]]

@@ Line 51: / Line 51: @@
-Enter the larger of the two&nbsp; $p$&ndash;values given by the equation &nbsp; $H_\text{bin}(p)= 0.5 \ \rm bit$&nbsp;.
+{Enter the larger of the two&nbsp; $p$&ndash;values given by the equation &nbsp; $H_\text{bin}(p)= 0.5 \ \rm bit$&nbsp;.
 |type="{}"}
 $p \ =  \ $ { 0.89 3% }

@@ Line 107: / Line 107: @@
-'''(5)'''&nbsp; Mit&nbsp; $p = 0.45$&nbsp; erhält man&nbsp; $H_{\rm bin}(p) = 0.993\hspace{0.05cm}\rm  bit$.&nbsp; Der relative Fehler bezüglich Entropie ist somit
+'''(5)'''&nbsp; With&nbsp; $p = 0.45$&nbsp; one obtains&nbsp; $H_{\rm bin}(p) = 0.993\hspace{0.05cm}\rm  bit$.&nbsp; The relative error with respect to entropy is thus
 :$$\varepsilon_{H} = \frac{H_{\rm bin}(p = 0.45)- H_{\rm bin}(p= 0.5)}{H_{\rm bin}(p = 0.5)}= \frac{0.993- 1}{1}\hspace{0.15cm}\underline {= -0.7 \, {\rm \%}}
 \hspace{0.05cm}.$$
-*Das Minuszeichen deutet darauf hin, dass der Entropiewert&nbsp; $H_{\rm bin}(p) = 0.993\hspace{0.05cm}\rm  bit$&nbsp; zu klein ist.
+*The minus sign indicates that the entropy value&nbsp; $H_{\rm bin}(p) = 0.993\hspace{0.05cm}\rm  bit$&nbsp; is too small.
-*Hätte die Simulation den zu großen Wert&nbsp; $p = 0.55$&nbsp; ergeben, so wäre die Entropie und auch der relative Fehler genau so groß.
+*If the simulation had yielded the too large value&nbsp; $p = 0.55$&nbsp;, the entropy and also the relative error would be just as large.
-'''(6)'''&nbsp; Es gilt&nbsp; $H_{\rm bin}(p = 0.045) = 0.265\hspace{0.05cm}\rm  bit$.
+'''(6)'''&nbsp; &nbsp; $H_{\rm bin}(p = 0.045) = 0.265\hspace{0.05cm}\rm  bit$ is valid..
-*Mit dem Ergebnis der Teilaufgabe&nbsp; '''(3)''' &nbsp; &#8658; &nbsp; $H_{\rm bin}(p = 0.05) = 0.286\hspace{0.05cm}\rm  bit$&nbsp; folgt daraus für den relativen Fehler bezüglich der Entropie:
+*With the result of subtask&nbsp; '''(3)''' &nbsp; &#8658; &nbsp; $H_{\rm bin}(p = 0.05) = 0.286\hspace{0.05cm}\rm  bit$&nbsp; it follows for the relative error with respect to the entropy:
 :$$\varepsilon_{H} = \frac{H_{\rm bin}(p = 0.045)- H_{\rm bin}(p= 0.05)}{H_{\rm bin}(p = 0.05)}= \frac{0.265- 0.286}{0.286}\hspace{0.15cm}\underline {= -7.3 \, {\rm \%}}
 \hspace{0.05cm}.$$
-*Das Ergebnis zeigt:
+*The result shows:
-#&nbsp;Eine falsche Bestimmung der Symbolwahrscheinlichkeiten um&nbsp; $10\%$&nbsp; macht sich für&nbsp; $p = 0.05$&nbsp; aufgrund des steileren&nbsp; $H_{\rm bin}(p)$&ndash;Verlaufs deutlich stärker bemerkbar als für&nbsp; $p = 0.5$.
+#&nbsp;An incorrect determination of the symbol probabilities by&nbsp; $10\%$&nbsp; is much more noticeable for&nbsp; $p = 0.05$&nbsp; due to the steeper&nbsp; $H_{\rm bin}(p)$&ndash;Verlaufs deutlich stärker bemerkbar als für&nbsp; $p = 0.5$.
-#&nbsp;Eine zu große Wahrscheinlichkeit&nbsp; $p = 0.055$&nbsp; hätte zu&nbsp; $H_{\rm bin}(p = 0.055) = 0.307\hspace{0.05cm}\rm  bit$&nbsp; geführt und damit zu einer Verfälschung um&nbsp; $\varepsilon_H = +7.3\%$.&nbsp; In diesem Bereich verläuft die Entropiekurve also (mit guter Näherung) linear.
+#&nbsp;A too large probability&nbsp; $p = 0.055$&nbsp; would have led to&nbsp; $H_{\rm bin}(p = 0.055) = 0.307\hspace{0.05cm}\rm  bit$&nbsp; and thus to a distortion of&nbsp; $\varepsilon_H = +7.3\%$.&nbsp; In this range, the entropy curve is thus linear (with a good approximation).
 {{ML-Fuß}}