Aufgaben:Exercise 1.3: Entropy Approximations - Revision history

Maintenance script: Fix interlanguage link: resolve redirect chain

2026-03-16T15:58:12Z

Fix interlanguage link: resolve redirect chain

← Older revision		Revision as of 17:58, 16 March 2026
Line 135:		Line 135:

	[[Category:Information Theory: Exercises\|^1.2 Sources with Memory^]]		[[Category:Information Theory: Exercises\|^1.2 Sources with Memory^]]
	[[de:Aufgaben:1.~~3_''H''_0,H1,H2,_..._,H~~]]		[[de:Aufgaben:Aufgabe 1.3: Entropienäherungen]]

Maintenance script: Join split $$ formula lines for correct rendering

2026-03-16T13:30:22Z

Join split $$ formula lines for correct rendering

← Older revision		Revision as of 15:30, 16 March 2026
Line 31:		Line 31:
	*This task belongs to the chapter  [[Information_Theory/Discrete_Sources_with_Memory\|Discrete Sources with Memory]].		*This task belongs to the chapter  [[Information_Theory/Discrete_Sources_with_Memory\|Discrete Sources with Memory]].
	*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:		*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:
	:$$H_k = \frac{1}{k} \cdot \sum_{i=1}^{2^k} p_i^{(k)} \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p_i^{(k)}} \hspace{0.5cm}({\rm unit\hspace{-0.1cm}: \hspace{0.1cm}bit/Symbol})		:$$H_k = \frac{1}{k} \cdot \sum_{i=1}^{2^k} p_i^{(k)} \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p_i^{(k)}} \hspace{0.5cm}({\rm unit\hspace{-0.1cm}: \hspace{0.1cm}bit/Symbol})\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$


Line 101:		Line 100:
	'''(4)'''  In the green symbol sequence  $($source  $\rm Q4)$ , the symbols  $\rm A$  and  $\rm B$  are equally likely:		'''(4)'''  In the green symbol sequence  $($source  $\rm Q4)$ , the symbols  $\rm A$  and  $\rm B$  are equally likely:
	[[File:P_ID2247__Inf_A_1_3d.png\|right\|frame\|Symbol sequences of a binary repetition code]]		[[File:P_ID2247__Inf_A_1_3d.png\|right\|frame\|Symbol sequences of a binary repetition code]]
	:$$p_{\rm A} = p_{\rm B} = 0.5 \hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_1 = 1\,{\rm bit/Symbol}		:$$p_{\rm A} = p_{\rm B} = 0.5 \hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_1 = 1\,{\rm bit/Symbol}\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$
	To determine  $H_2$, one considers two-tuples.  The composite probabilities  $p_{\rm AA}$,  $p_{\rm AB}$,  $p_{\rm BA}$  and  $p_{\rm BB}$  can be calculated from this.  You can see from the sketch:		To determine  $H_2$, one considers two-tuples.  The composite probabilities  $p_{\rm AA}$,  $p_{\rm AB}$,  $p_{\rm BA}$  and  $p_{\rm BB}$  can be calculated from this.  You can see from the sketch:
	* The combinations  $\rm AB$  and  $\rm BA$  are only possible if a tuple starts at even  $\nu$ .  For the composite probabilities  $p_{\rm AB}$  and  $p_{\rm BA}$  then holds:		* The combinations  $\rm AB$  and  $\rm BA$  are only possible if a tuple starts at even  $\nu$ .  For the composite probabilities  $p_{\rm AB}$  and  $p_{\rm BA}$  then holds:
	:$$p_{\rm AB} \hspace{0.1cm} = \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}is\hspace{0.15cm}even}) \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} \| q_{\nu} = \mathbf{A}) = {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}		:$$p_{\rm AB} \hspace{0.1cm} = \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}is\hspace{0.15cm}even}) \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} \| q_{\nu} = \mathbf{A}) = {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$
	*In contrast, for the two other combinations  $\rm AA$  and  $\rm BB$:		*In contrast, for the two other combinations  $\rm AA$  and  $\rm BB$:
	:$$p_{\rm AA} ={\rm Pr}(\nu = 1) \cdot {\rm Pr}( q_1 = \mathbf{A}) \cdot {\rm Pr}(q_{2} = \mathbf{A}\hspace{0.05cm} \| q_{1} = \mathbf{A}) + {\rm Pr}(\nu=2) \cdot {\rm Pr}( q_{2} = \mathbf{A}) \cdot {\rm Pr}(q_{3} = \mathbf{A}\hspace{0.05cm} \| q_{2} = \mathbf{A})		:$$p_{\rm AA} ={\rm Pr}(\nu = 1) \cdot {\rm Pr}( q_1 = \mathbf{A}) \cdot {\rm Pr}(q_{2} = \mathbf{A}\hspace{0.05cm} \| q_{1} = \mathbf{A}) + {\rm Pr}(\nu=2) \cdot {\rm Pr}( q_{2} = \mathbf{A}) \cdot {\rm Pr}(q_{3} = \mathbf{A}\hspace{0.05cm} \| q_{2} = \mathbf{A})\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$		:$$\Rightarrow \hspace{0.3cm}p_{\rm AA} = \frac{1}{2} \cdot \frac{1}{2} \cdot 1+ \frac{1}{2} \cdot \frac{1}{2} \cdot \frac{1}{2} = \frac{3}{8} = p_{\rm BB}\hspace{0.05cm}.$$
	:$$\Rightarrow \hspace{0.3cm}p_{\rm AA} = \frac{1}{2} \cdot \frac{1}{2} \cdot 1+ \frac{1}{2} \cdot \frac{1}{2} \cdot \frac{1}{2} = \frac{3}{8} = p_{\rm BB}
	\hspace{0.05cm}.$$
	:Here  $\nu = 1$  stands for all odd indices and  $\nu = 2$  for all even indices.		:Here  $\nu = 1$  stands for all odd indices and  $\nu = 2$  for all even indices.

	*This gives for the entropy approximation:		*This gives for the entropy approximation:
	:$$H_2 = \frac{1}{2} \cdot \left [ 2 \cdot \frac{3}{8} \cdot {\rm log}_2\hspace{0.1cm}\frac {8}{3} +		:$$H_2 = \frac{1}{2} \cdot \left [ 2 \cdot \frac{3}{8} \cdot {\rm log}_2\hspace{0.1cm}\frac {8}{3} +2 \cdot \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8)\right ] =\frac{3}{8} \cdot{\rm log}_2\hspace{0.1cm}(8) - \frac{3}{8} \cdot {\rm log}_2\hspace{0.1cm}(3) + \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8) \hspace{0.15cm} \underline {= 0.906 \,{\rm bit/symbol}}\hspace{0.05cm}.$$
	2 \cdot \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8)\right ] =
	\frac{3}{8} \cdot
	{\rm log}_2\hspace{0.1cm}(8) - \frac{3}{8} \cdot {\rm log}_2\hspace{0.1cm}(3) + \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8) \hspace{0.15cm} \underline {= 0.906 \,{\rm bit/symbol}}
	\hspace{0.05cm}.$$


Line 126:		Line 117:
	:$$p_{\rm AAA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm BBB} = 1/4 \hspace{0.05cm},\hspace{0.2cm} p_{\rm ABA} = p_{\rm BAB} = 0 \hspace{0.05cm},\hspace{0.2cm} p_{\rm AAB} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABB} = p_{\rm BBA} = p_{\rm BAA} = 1/8$$		:$$p_{\rm AAA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm BBB} = 1/4 \hspace{0.05cm},\hspace{0.2cm} p_{\rm ABA} = p_{\rm BAB} = 0 \hspace{0.05cm},\hspace{0.2cm} p_{\rm AAB} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABB} = p_{\rm BBA} = p_{\rm BAA} = 1/8$$
	and from this to the entropy approximation		and from this to the entropy approximation
	:$$H_3 = \frac{1}{3} \cdot \left [ 2 \cdot \frac{1}{4} \cdot {\rm log}_2\hspace{0.1cm}(4) +		:$$H_3 = \frac{1}{3} \cdot \left [ 2 \cdot \frac{1}{4} \cdot {\rm log}_2\hspace{0.1cm}(4) +4 \cdot \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8)\right ] = \frac{2.5}{3} \hspace{0.15cm} \underline {= 0.833 \,{\rm bit/Symbol}}\hspace{0.05cm}.$$
	4 \cdot \frac{1}{8} \cdot {\rm log}_2\hspace{0.1cm}(8)\right ] = \frac{2.5}{3} \hspace{0.15cm} \underline {= 0.833 \,{\rm bit/Symbol}}
	\hspace{0.05cm}.$$
	To calculate  $H_4$ , the  $16$  probabilities are as follows:		To calculate  $H_4$ , the  $16$  probabilities are as follows:
	:$$p_{\rm AAAA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm BBBB} = 3/16 \hspace{0.05cm},\hspace{0.2cm} p_{\rm AABB} = p_{\rm BBAA} = 2/16 \hspace{0.05cm},$$		:$$p_{\rm AAAA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm BBBB} = 3/16 \hspace{0.05cm},\hspace{0.2cm} p_{\rm AABB} = p_{\rm BBAA} = 2/16 \hspace{0.05cm},$$
	:$$ p_{\rm AAAB} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABBA} = p_{\rm ABBB} = p_{\rm BBBA} = p_{\rm BAAB} = p_{\rm BAAA}= 1/16		:$$ p_{\rm AAAB} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABBA} = p_{\rm ABBB} = p_{\rm BBBA} = p_{\rm BAAB} = p_{\rm BAAA}= 1/16\hspace{0.05cm}$$
	\hspace{0.05cm}$$
	:$$ p_{\rm AABA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABAA} = p_{\rm ABAB} = p_{\rm BBAB} = p_{\rm BABB} = p_{\rm BABA}= 0\hspace{0.05cm}.$$		:$$ p_{\rm AABA} \hspace{0.1cm} = \hspace{0.1cm} p_{\rm ABAA} = p_{\rm ABAB} = p_{\rm BBAB} = p_{\rm BABB} = p_{\rm BABA}= 0\hspace{0.05cm}.$$
	It follows that:		It follows that:
	:$$H_4= \frac{1}{4} \hspace{-0.05cm}\cdot \hspace{-0.05cm}\left [ 2 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{3}{16} \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.1cm}\frac{16}{3} +		:$$H_4= \frac{1}{4} \hspace{-0.05cm}\cdot \hspace{-0.05cm}\left [ 2 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{3}{16} \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.1cm}\frac{16}{3} +2 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{1}{8} \hspace{-0.05cm}\cdot \hspace{-0.05cm}{\rm log}_2\hspace{0.1cm}(8) +6 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{1}{16} \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.1cm}(16)\right ] =\frac{\left [ 6 \hspace{-0.05cm}\cdot \hspace{-0.05cm}{\rm log}_2\hspace{0.01cm}(16) - 6 \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.01cm}(3) + 4 \hspace{-0.05cm}\cdot \hspace{-0.05cm}{\rm log}_2\hspace{0.01cm}(8) + 6\hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.01cm}(16)\right ]}{32} .$$
	2 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{1}{8} \hspace{-0.05cm}\cdot \hspace{-0.05cm}{\rm log}_2\hspace{0.1cm}(8) +
	6 \hspace{-0.05cm}\cdot \hspace{-0.05cm} \frac{1}{16} \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.1cm}(16)\right ] =\frac{\left [ 6 \hspace{-0.05cm}\cdot \hspace{-0.05cm}
	{\rm log}_2\hspace{0.01cm}(16) - 6 \hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.01cm}(3) + 4 \hspace{-0.05cm}\cdot \hspace{-0.05cm}
	{\rm log}_2\hspace{0.01cm}(8) + 6\hspace{-0.05cm}\cdot \hspace{-0.05cm} {\rm log}_2\hspace{0.01cm}(16)\right ]}{32} .$$
	One can see:		One can see:
	*Even the approximation  $H_4 = 0.789\,{\rm bit/Symbol}$  still deviates significantly from the final entropy value  $H = 0.5\,{\rm bit/symbol}$ .		*Even the approximation  $H_4 = 0.789\,{\rm bit/Symbol}$  still deviates significantly from the final entropy value  $H = 0.5\,{\rm bit/symbol}$ .
	*The repetition code obviously cannot be modelled by a Markov source.  If  $\rm Q4$  were a Markov source, then the following would have to apply:		*The repetition code obviously cannot be modelled by a Markov source.  If  $\rm Q4$  were a Markov source, then the following would have to apply:
	:$$H = 2 \cdot H_2 - H_1		:$$H = 2 \cdot H_2 - H_1\hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_2 = 1/2 \cdot (H+H_1) =1/2 \cdot (0.5+1) = 0.75 \,{\rm bit/Symbol}\hspace{0.05cm}.$$
	\hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_2 = 1/2 \cdot (H+H_1) =
	1/2 \cdot (0.5+1) = 0.75 \,{\rm bit/Symbol}\hspace{0.05cm}.$$

Maintenance script: Add German interlanguage link

2026-03-16T12:30:21Z

Add German interlanguage link

← Older revision		Revision as of 14:30, 16 March 2026
Line 153:		Line 153:

	[[Category:Information Theory: Exercises\|^1.2 Sources with Memory^]]		[[Category:Information Theory: Exercises\|^1.2 Sources with Memory^]]
			[[de:Aufgaben:1.3_''H''_0,H1,H2,_..._,H]]

Guenter at 12:02, 10 August 2021

2021-08-10T12:02:03Z

← Older revision		Revision as of 14:02, 10 August 2021
Line 1:		Line 1:

	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Nachrichtenquellen mit Gedächtnis~~		{{quiz-Header\|Buchseite=Information_Theory/Discrete_Sources_with_Memory
	}}		}}

Guenter at 11:42, 30 June 2021

2021-06-30T11:42:03Z

← Older revision		Revision as of 13:42, 30 June 2021
Line 3:		Line 3:
	}}		}}

	[[File:~~Inf_A_1_3_vers2~~.png\|right\|frame\|Different binary sequences]]		[[File:EN_Inf_A_1_3_v2.png\|right\|frame\|Different binary sequences]]
	The graphic on the right shows four symbol sequences  $\langle q_\nu \rangle $,  each with length  $N = 60$.  The source symbols are  $\rm A$  and  $\rm B$.		The graphic on the right shows four symbol sequences  $\langle q_\nu \rangle $,  each with length  $N = 60$.  The source symbols are  $\rm A$  and  $\rm B$.
	*It follows directly that  $H_0 = 1 \; \rm bit/symbol$  applies to the decision content of all sources considered.		*It follows directly that  $H_0 = 1 \; \rm bit/symbol$  applies to the decision content of all sources considered.
Line 26:		Line 26:
	Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for the source  $\rm Q4$ .		Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for the source  $\rm Q4$ .

	[[File:~~Inf_A_1_3b_vers2~~.png\|left\|frame\|Source entropy and approximations in "bit/symbol"]]		[[File:EN_Inf_A_1_3b_v2.png\|left\|frame\|Source entropy and approximations in "bit/symbol"]]
	<br clear=all>		<br clear=all>
	''Hints:''		''Hints:''

Guenter at 14:02, 19 June 2021

2021-06-19T14:02:00Z

← Older revision		Revision as of 16:02, 19 June 2021
Line 29:		Line 29:
	<br clear=all>		<br clear=all>
	''Hints:''		''Hints:''
	*This task belongs to the chapter  [[Information_Theory/Discrete_Sources_with_Memory\|Sources with Memory]].		*This task belongs to the chapter  [[Information_Theory/Discrete_Sources_with_Memory\|Discrete Sources with Memory]].

	*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:		*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:
	:$$H_k = \frac{1}{k} \cdot \sum_{i=1}^{2^k} p_i^{(k)} \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p_i^{(k)}} \hspace{0.5cm}({\rm unit\hspace{-0.1cm}: \hspace{0.1cm}bit/Symbol})		:$$H_k = \frac{1}{k} \cdot \sum_{i=1}^{2^k} p_i^{(k)} \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p_i^{(k)}} \hspace{0.5cm}({\rm unit\hspace{-0.1cm}: \hspace{0.1cm}bit/Symbol})

Guenter at 13:39, 19 June 2021

2021-06-19T13:39:41Z

← Older revision		Revision as of 15:39, 19 June 2021
Line 29:		Line 29:
	<br clear=all>		<br clear=all>
	''Hints:''		''Hints:''
	*This task belongs to the chapter  [[Information_Theory/~~Nachrichtenquellen_mit_Gedächtnis~~\|Sources with Memory]].		*This task belongs to the chapter  [[Information_Theory/Discrete_Sources_with_Memory\|Sources with Memory]].

	*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:		*For the  $k$–th entropy approximation, the following holds for binary sources  $(M = 2)$  with the composite probability  $ p_i^{(k)}$  of a  $k$–tuple:

Guenter at 13:03, 19 June 2021

2021-06-19T13:03:37Z

Guenter at 12:55, 19 June 2021

2021-06-19T12:55:17Z

← Older revision		Revision as of 14:55, 19 June 2021
Line 4:		Line 4:

	[[File:Inf_A_1_3_vers2.png\|right\|frame\|Different binary sequences]]		[[File:Inf_A_1_3_vers2.png\|right\|frame\|Different binary sequences]]
	The ~~diagram~~ shows four symbol sequences  $\langle q_\nu \rangle $ , each with length  $N = 60$.  The source symbols are  $\rm A$  and  $\rm B$ ~~respectively~~.		The graphic on the right shows four symbol sequences  $\langle q_\nu \rangle $,  each with length  $N = 60$.  The source symbols are  $\rm A$  and  $\rm B$.
	*It follows directly that  $H_0 = 1 \; \rm bit/symbol$  applies to the decision content of all sources considered.		*It follows directly that  $H_0 = 1 \; \rm bit/symbol$  applies to the decision content of all sources considered.
	*However, the symbols  $\rm A$  and  $\rm B$  do not occur with equal probability, but with the probabilities  $p_{\rm A}$  and  $p_{\rm B}$.		*However, the symbols  $\rm A$  and  $\rm B$  do not occur with equal probability, but with the probabilities  $p_{\rm A}$  and  $p_{\rm B}$.
Line 19:		Line 19:
	The following size relations exist between these entropies:   $H \le$ ... $\le H_3 \le H_2 \le H_1 \le H_0 \hspace{0.05cm}.$		The following size relations exist between these entropies:   $H \le$ ... $\le H_3 \le H_2 \le H_1 \le H_0 \hspace{0.05cm}.$

	*What is not known is the correlation between the sources  $\rm Q1$,  $\rm Q2$,  $\rm Q3$,  $\rm Q4$  and the symbol sequences shown in the graph  (black, blue, red, green).		*What is not known is the correlation between the sources  $\rm Q1$,  $\rm Q2$,  $\rm Q3$,  $\rm Q4$  and the symbol sequences shown in the graph  <br>(black, blue, red, green).
	*It is only known that source  $\rm Q4$  contains a repetition code.  Due to the fact that in the corresponding symbol sequence every second symbol does not lier any information,  $~~H_0~~ = 0.5 \; \rm bit/symbol$.		*It is only known that source  $\rm Q4$  contains a repetition code.  Due to the fact that in the corresponding symbol sequence every second symbol does not lier any information,  the final entrpy value is  $H = 0.5 \; \rm bit/symbol$.
	*In addition, the entropy approximations  $H_1 = 1 \; \rm bit/symbol$~~  (equally probable symbols)~~  and  $H_4 \approx 0.789 \; \rm bit/symbol$  are given.		*In addition, the entropy approximations  $H_1 = 1 \; \rm bit/symbol$  and  $H_4 \approx 0.789 \; \rm bit/symbol$  are given.


	Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for ~~this news~~ source  $\rm Q4$ .		Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for the source  $\rm Q4$ .

	[[File:Inf_A_1_3b_vers2.png\|left\|frame\|Source entropy and approximations in "bit/symbol"]]		[[File:Inf_A_1_3b_vers2.png\|left\|frame\|Source entropy and approximations in "bit/symbol"]]

Javier: Text replacement - "”" to """

2021-05-28T14:41:12Z

Text replacement - "”" to """

← Older revision		Revision as of 16:41, 28 May 2021
Line 26:		Line 26:
	Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for this news source  $\rm Q4$ .		Finally, the entropy approximations  $H_2$  and  $H_3$ are to be determined for this news source  $\rm Q4$ .

	[[File:Inf_A_1_3b_vers2.png\|left\|frame\|Source entropy and approximations in "bit/symbol~~”~~]]		[[File:Inf_A_1_3b_vers2.png\|left\|frame\|Source entropy and approximations in "bit/symbol"]]
	<br clear=all>		<br clear=all>
	''Hints:''		''Hints:''

← Older revision		Revision as of 15:03, 19 June 2021
Line 79:		Line 79:
	{{ML-Kopf}}		{{ML-Kopf}}
	'''(1)'''  The black binary sequence comes from the source  $\underline{\rm Q3}$,		'''(1)'''  The black binary sequence comes from the source  $\underline{\rm Q3}$,
	*since the symbols are equally probable   ⇒   $H_1 = H_0$ and		*since the symbols are equally probable   ⇒   $H_1 = H_0$,  and
	*there are no statistical ~~ties~~ between the symbols   ⇒   $H=$ ... $= H_2 = H_1$.		*there are no statistical bindings between the symbols   ⇒   $H=$ ... $= H_2 = H_1$.


Line 94:		Line 94:
	*The red symbol sequence has the following properties:		*The red symbol sequence has the following properties:
	:* Because of  $H_1 = H_0$ , the symbols are equally probable:   $p_{\rm A} = p_{\rm B} = 0.5$.		:* Because of  $H_1 = H_0$ , the symbols are equally probable:   $p_{\rm A} = p_{\rm B} = 0.5$.
	:* Because of  $H < H_1$  , there are statistical ~~bonds~~ within the sequence.		:* Because of  $H < H_1$,  there are statistical bindings within the sequence.
	*This can be recognised by the fact that there are more transitions between  $\rm A$  and  $\rm B$  than with statistical independence.		*This can be recognised by the fact that there are more transitions between  $\rm A$  and  $\rm B$  than with statistical independence.

Line 100:		Line 100:


	'''(4)'''  In the green symbol sequence  $($source $\rm Q4)$ , the symbols  $\rm A$  and  $\rm B$  are equally likely:		'''(4)'''  In the green symbol sequence  $($source  $\rm Q4)$ , the symbols  $\rm A$  and  $\rm B$  are equally likely:
	[[File:P_ID2247__Inf_A_1_3d.png\|right\|frame\|Symbol sequences of a binary repetition code]]		[[File:P_ID2247__Inf_A_1_3d.png\|right\|frame\|Symbol sequences of a binary repetition code]]
	:$$p_{\rm A} = p_{\rm B} = 0.5 \hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_1 = 1\,{\rm bit/Symbol}		:$$p_{\rm A} = p_{\rm B} = 0.5 \hspace{0.3cm}\Rightarrow\hspace{0.3cm}H_1 = 1\,{\rm bit/Symbol}
Line 106:		Line 106:
	To determine  $H_2$, one considers two-tuples.  The composite probabilities  $p_{\rm AA}$,  $p_{\rm AB}$,  $p_{\rm BA}$  and  $p_{\rm BB}$  can be calculated from this.  You can see from the sketch:		To determine  $H_2$, one considers two-tuples.  The composite probabilities  $p_{\rm AA}$,  $p_{\rm AB}$,  $p_{\rm BA}$  and  $p_{\rm BB}$  can be calculated from this.  You can see from the sketch:
	* The combinations  $\rm AB$  and  $\rm BA$  are only possible if a tuple starts at even  $\nu$ .  For the composite probabilities  $p_{\rm AB}$  and  $p_{\rm BA}$  then holds:		* The combinations  $\rm AB$  and  $\rm BA$  are only possible if a tuple starts at even  $\nu$ .  For the composite probabilities  $p_{\rm AB}$  and  $p_{\rm BA}$  then holds:
	:$$p_{\rm AB} \hspace{0.1cm} = \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}~~ist~~\hspace{0.15cm}~~gerade~~}) \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} \| q_{\nu} = \mathbf{A}) = {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}		:$$p_{\rm AB} \hspace{0.1cm} = \hspace{0.1cm} {\rm Pr}(\nu {\rm \hspace{0.15cm}is\hspace{0.15cm}even}) \cdot {\rm Pr}( q_{\nu} = \mathbf{A}) \cdot {\rm Pr}(q_{\nu+1} = \mathbf{B}\hspace{0.05cm} \| q_{\nu} = \mathbf{A}) = {1}/{2} \cdot {1}/{2} \cdot {1}/{2} = {1}/{8} = p_{\rm BA}
	\hspace{0.05cm}.$$		\hspace{0.05cm}.$$
	*In contrast, for the two other combinations  $\rm AA$  and  $\rm BB$:		*In contrast, for the two other combinations  $\rm AA$  and  $\rm BB$: