Aufgaben:Exercise 1.3: Entropy Approximations - Revision history

Guenter at 12:02, 10 August 2021

2021-08-10T12:02:03Z

Guenter at 11:42, 30 June 2021

2021-06-30T11:42:03Z

Guenter at 14:02, 19 June 2021

2021-06-19T14:02:00Z

Guenter at 13:39, 19 June 2021

2021-06-19T13:39:41Z

Guenter at 13:03, 19 June 2021

2021-06-19T13:03:37Z

Guenter at 12:55, 19 June 2021

2021-06-19T12:55:17Z

Javier: Text replacement - "”" to """

2021-05-28T14:41:12Z

Text replacement - "”" to """

Javier: Text replacement - "„" to """

2021-05-28T14:34:07Z

Text replacement - "„" to """

Guenter at 15:45, 25 May 2021

2021-05-25T15:45:12Z

Guenter: Guenter moved page Aufgaben:Aufgabe 1.3: Entropienäherungen to Aufgaben:Exercise 1.3: Entropy Approximations

2021-05-25T15:44:16Z

Guenter moved page Aufgabe 1.3: Entropienäherungen to Exercise 1.3: Entropy Approximations

← Older revision		Revision as of 12:02, 10 August 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Nachrichtenquellen mit Gedächtnis~~	+	{{quiz-Header\|Buchseite=Information_Theory/Discrete_Sources_with_Memory
	}}		}}

@@ Line 3: / Line 3: @@
 }}
-[[File:Inf_A_1_3_vers2.png|right|frame|Different binary sequences]]
+[[File:EN_Inf_A_1_3_v2.png|right|frame|Different binary sequences]]
 The graphic on the right shows four symbol sequences&nbsp; $\langle q_\nu \rangle $,&nbsp; each with length&nbsp; $N = 60$.&nbsp; The source symbols are&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$.
 *It follows directly that&nbsp; $H_0 = 1 \; \rm bit/symbol$&nbsp; applies to the decision content of all sources considered.
@@ Line 26: / Line 26: @@
 Finally, the entropy approximations&nbsp; $H_2$&nbsp; and&nbsp; $H_3$ are to be determined for the source&nbsp; $\rm Q4$&nbsp;.
-[[File:Inf_A_1_3b_vers2.png|left|frame|Source entropy and approximations in "bit/symbol"]]
+[[File:EN_Inf_A_1_3b_v2.png|left|frame|Source entropy and approximations in "bit/symbol"]]
 <br clear=all>
 ''Hints:''

@@ Line 29: / Line 29: @@
 <br clear=all>
 ''Hints:''
-*This task belongs to the chapter&nbsp; [[Information_Theory/Discrete_Sources_with_Memory|Sources with Memory]].
+*This task belongs to the chapter&nbsp; [[Information_Theory/Discrete_Sources_with_Memory|Discrete Sources with Memory]].
 *For the&nbsp; $k$&ndash;th entropy approximation, the following holds for binary sources&nbsp; $(M = 2)$&nbsp; with the composite probability&nbsp; $ p_i^{(k)}$&nbsp; of a&nbsp; $k$&ndash;tuple:
 :$$H_k = \frac{1}{k} \cdot \sum_{i=1}^{2^k} p_i^{(k)} \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p_i^{(k)}} \hspace{0.5cm}({\rm unit\hspace{-0.1cm}: \hspace{0.1cm}bit/Symbol})

@@ Line 29: / Line 29: @@
 <br clear=all>
 ''Hints:''
-*This task belongs to the chapter&nbsp; [[Information_Theory/Nachrichtenquellen_mit_Gedächtnis|Sources with Memory]].
+*This task belongs to the chapter&nbsp; [[Information_Theory/Discrete_Sources_with_Memory|Sources with Memory]].
 *For the&nbsp; $k$&ndash;th entropy approximation, the following holds for binary sources&nbsp; $(M = 2)$&nbsp; with the composite probability&nbsp; $ p_i^{(k)}$&nbsp; of a&nbsp; $k$&ndash;tuple:

@@ Line 79: / Line 79: @@
 {{ML-Kopf}}
 '''(1)'''&nbsp; The black binary sequence comes from the source&nbsp; $\underline{\rm Q3}$,
-*since the symbols are equally probable &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H_1 = H_0$ and
+*since the symbols are equally probable &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H_1 = H_0$,&nbsp; and
-*there are no statistical ties between the symbols &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H=$ ...  $= H_2 = H_1$.
+*there are no statistical bindings between the symbols &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H=$ ...  $= H_2 = H_1$.
@@ Line 94: / Line 94: @@
 *The red symbol sequence has the following properties:
 :* Because of&nbsp; $H_1 = H_0$&nbsp;, the symbols are equally probable:  &nbsp; $p_{\rm A} = p_{\rm B} = 0.5$.
-:* Because of&nbsp; $H < H_1$&nbsp; , there are statistical bonds within the sequence.
+:* Because of&nbsp; $H < H_1$,&nbsp; there are statistical bindings within the sequence.
 *This can be recognised by the fact that there are more transitions between&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$&nbsp; than with statistical independence.
@@ Line 100: / Line 100: @@
-'''(4)'''&nbsp; In the green symbol sequence&nbsp; $($source $\rm Q4)$&nbsp;, the symbols&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$&nbsp; are equally likely:
+'''(4)'''&nbsp; In the green symbol sequence&nbsp; $($source&nbsp; $\rm Q4)$&nbsp;, the symbols&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$&nbsp; are equally likely:
 [[File:P_ID2247__Inf_A_1_3d.png|right|frame|Symbol sequences of a binary repetition code]]
 :$$p_{\rm A} = p_{\rm B} = 0.5 \hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_1 = 1\,{\rm bit/Symbol}
@@ Line 106: / Line 106: @@
 To determine&nbsp; $H_2$, one considers two-tuples.&nbsp; The composite probabilities&nbsp; $p_{\rm AA}$,&nbsp; $p_{\rm AB}$,&nbsp; $p_{\rm BA}$&nbsp; and&nbsp; $p_{\rm BB}$&nbsp; can be calculated from this.&nbsp; You can see from the sketch:
 * The combinations&nbsp; $\rm AB$&nbsp; and&nbsp; $\rm BA$&nbsp; are only possible if a tuple starts at even&nbsp; $\nu$&nbsp;.&nbsp; For the composite probabilities&nbsp; $p_{\rm AB}$&nbsp; and&nbsp; $p_{\rm BA}$&nbsp; then holds:
-:$$p_{\rm AB} \hspace{0.1cm} =  \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}ist\hspace{0.15cm}gerade})  \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} | q_{\nu} = \mathbf{A}) =  {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}
+:$$p_{\rm AB} \hspace{0.1cm} =  \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}is\hspace{0.15cm}even})  \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} | q_{\nu} = \mathbf{A}) =  {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}
   \hspace{0.05cm}.$$
 *In contrast, for the two other combinations&nbsp; $\rm AA$&nbsp; and&nbsp; $\rm BB$:

@@ Line 4: / Line 4: @@
 [[File:Inf_A_1_3_vers2.png|right|frame|Different binary sequences]]
-The diagram shows four symbol sequences&nbsp; $\langle q_\nu \rangle $&nbsp;, each with length&nbsp; $N = 60$.&nbsp; The source symbols are&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$ respectively.
+The graphic on the right shows four symbol sequences&nbsp; $\langle q_\nu \rangle $,&nbsp; each with length&nbsp; $N = 60$.&nbsp; The source symbols are&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$.
 *It follows directly that&nbsp; $H_0 = 1 \; \rm bit/symbol$&nbsp; applies to the decision content of all sources considered.
 *However, the symbols&nbsp; $\rm A$&nbsp; and&nbsp; $\rm B$&nbsp; do not occur with equal probability, but with the probabilities&nbsp; $p_{\rm A}$&nbsp; and&nbsp; $p_{\rm B}$.
@@ Line 19: / Line 19: @@
 The following size relations exist between these entropies: &nbsp; $H \le$ ... $\le H_3 \le H_2 \le H_1 \le H_0 \hspace{0.05cm}.$
-*What is not known is the correlation between the sources&nbsp; $\rm Q1$,&nbsp; $\rm Q2$,&nbsp; $\rm Q3$,&nbsp; $\rm Q4$&nbsp; and the symbol sequences shown in the graph&nbsp; (black, blue, red, green).
+*What is not known is the correlation between the sources&nbsp; $\rm Q1$,&nbsp; $\rm Q2$,&nbsp; $\rm Q3$,&nbsp; $\rm Q4$&nbsp; and the symbol sequences shown in the graph&nbsp; <br>(black, blue, red, green).
-*It is only known that source&nbsp; $\rm Q4$&nbsp; contains a repetition code.&nbsp; Due to the fact that in the corresponding symbol sequence every second symbol does not lier any information,&nbsp; $H_0 = 0.5 \; \rm bit/symbol$.
+*It is only known that source&nbsp; $\rm Q4$&nbsp; contains a repetition code.&nbsp; Due to the fact that in the corresponding symbol sequence every second symbol does not lier any information,&nbsp; the final entrpy value is&nbsp; $H = 0.5 \; \rm bit/symbol$.
-*In addition, the entropy approximations&nbsp; $H_1 = 1 \; \rm bit/symbol$&nbsp; (equally probable symbols)&nbsp; and&nbsp; $H_4 \approx 0.789 \; \rm bit/symbol$&nbsp; are given.
+*In addition, the entropy approximations&nbsp; $H_1 = 1 \; \rm bit/symbol$&nbsp; and&nbsp; $H_4 \approx 0.789 \; \rm bit/symbol$&nbsp; are given.
-Finally, the entropy approximations&nbsp; $H_2$&nbsp; and&nbsp; $H_3$ are to be determined for this news source&nbsp; $\rm Q4$&nbsp;.
+Finally, the entropy approximations&nbsp; $H_2$&nbsp; and&nbsp; $H_3$ are to be determined for the source&nbsp; $\rm Q4$&nbsp;.
 [[File:Inf_A_1_3b_vers2.png|left|frame|Source entropy and approximations in "bit/symbol"]]

← Older revision		Revision as of 15:45, 25 May 2021
Line 153:		Line 153:


−	[[Category:Information Theory: Exercises\|^1.2 ~~Nachrichtenquellen mit Gedächtnis~~^]]	+	[[Category:Information Theory: Exercises\|^1.2 Sources with Memory^]]

← Older revision	Revision as of 15:44, 25 May 2021
(No difference)