Aufgaben:Exercise 1.2: Entropy of Ternary Sources - Revision history

Bene: Text replacement - "autocorrelation" to "auto-correlation"

2022-02-17T11:29:04Z

Text replacement - "autocorrelation" to "auto-correlation"

Javier: Text replacement - "discrete-value" to "discrete"

2021-10-11T08:28:10Z

Text replacement - "discrete-value" to "discrete"

Guenter at 12:01, 10 August 2021

2021-08-10T12:01:04Z

Guenter at 12:00, 10 August 2021

2021-08-10T12:00:36Z

Guenter at 15:25, 16 June 2021

2021-06-16T15:25:31Z

Guenter at 15:10, 16 June 2021

2021-06-16T15:10:30Z

Javier: Text replacement - "„" to """

2021-05-28T14:41:44Z

Text replacement - "„" to """

Javier: Text replacement - "”" to """

2021-05-28T14:39:58Z

Text replacement - "”" to """

Guenter at 12:34, 13 May 2021

2021-05-13T12:34:42Z

Noah at 16:40, 8 May 2021

2021-05-08T16:40:14Z

@@ Line 85: / Line 85: @@
 *It does not matter which numerical values or physical quantities one assigns to the individual symbols.
 *It is different with mean values or the ACF (auto correlation function) calculation.&nbsp; If only symbols are given, no moments can be calculated for them.
-*Moreover, the mean values, autocorrelation, etc. depend on whether one agrees on the assignment bipolar&nbsp; $(-1, \hspace{0.10cm}0, \hspace{0.05cm}+1)$&nbsp;  or unipolar&nbsp;  $(0, \hspace{0.05cm}1, \hspace{0.05cm}2)$&nbsp;.
+*Moreover, the mean values, auto-correlation, etc. depend on whether one agrees on the assignment bipolar&nbsp; $(-1, \hspace{0.10cm}0, \hspace{0.05cm}+1)$&nbsp;  or unipolar&nbsp;  $(0, \hspace{0.05cm}1, \hspace{0.05cm}2)$&nbsp;.

@@ Line 3: / Line 3: @@
 [[File:Inf_A_1_2_vers2.png|right|frame|Probabilities of two ternary sources]]
-The entropy of a discrete-value memoryless source with&nbsp; $M$&nbsp; possible symbols is:
+The entropy of a discrete memoryless source with&nbsp; $M$&nbsp; possible symbols is:
 :$$H =  \sum_{\mu = 1}^M p_{\mu} \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{p_\mu}\hspace{0.05cm},\hspace{0.3cm}
   {\rm pseudo unit\hspace{-0.15cm}: \hspace{0.15cm}bit}\hspace{0.05cm}.$$

← Older revision		Revision as of 12:01, 10 August 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=Information_Theory/Discrete_Memoryless_Sources	+	{{quiz-Header\|Buchseite=Information_Theory/Discrete_Memoryless_Sources}}

	[[File:Inf_A_1_2_vers2.png\|right\|frame\|Probabilities of two ternary sources]]		[[File:Inf_A_1_2_vers2.png\|right\|frame\|Probabilities of two ternary sources]]

← Older revision		Revision as of 12:00, 10 August 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Gedächtnislose Nachrichtenquellen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Discrete_Memoryless_Sources
−	}}

	[[File:Inf_A_1_2_vers2.png\|right\|frame\|Probabilities of two ternary sources]]		[[File:Inf_A_1_2_vers2.png\|right\|frame\|Probabilities of two ternary sources]]

@@ Line 77: / Line 77: @@
 ===Solution===
 {{ML-Kopf}}
-'''(1)'''&nbsp; With the probabilities of occurence&nbsp; $1/2$,&nbsp; $1/3$&nbsp; and&nbsp; $1/6$&nbsp; we get the following entropy value:
+'''(1)'''&nbsp; With the "symbol" probabilities&nbsp; $1/2$,&nbsp; $1/3$&nbsp; and&nbsp; $1/6$&nbsp; we get the following entropy value:
 :$$H \hspace{0.1cm}  =  \hspace{0.1cm}  1/2 \cdot {\rm log}_2\hspace{0.1cm}(2) +1/3 \cdot {\rm log}_2\hspace{0.1cm}(3) +1/6 \cdot {\rm log}_2\hspace{0.1cm}(6) =(1/2 + 1/6)\cdot {\rm log}_2\hspace{0.1cm}(2) +  (1/3 + 1/6)\cdot {\rm log}_2\hspace{0.1cm}(3) \hspace{0.15cm}\underline {\approx 1.46 \, {\rm bit}} \hspace{0.05cm}.$$
@@ Line 85: / Line 85: @@
 *The entropy depends only on the probabilities of occurrence.
 *It does not matter which numerical values or physical quantities one assigns to the individual symbols.
-*It is different with mean values or the ACF (auto correlation function)calculation&nbsp; If only symbols are given, no moments can be given for them.
+*It is different with mean values or the ACF (auto correlation function) calculation.&nbsp; If only symbols are given, no moments can be calculated for them.
 *Moreover, the mean values, autocorrelation, etc. depend on whether one agrees on the assignment bipolar&nbsp; $(-1, \hspace{0.10cm}0, \hspace{0.05cm}+1)$&nbsp;  or unipolar&nbsp;  $(0, \hspace{0.05cm}1, \hspace{0.05cm}2)$&nbsp;.
@@ Line 96: / Line 96: @@
-'''(4)'''&nbsp; The maximum entropy of a memoryless source with symbol range $M$ is obtained when all&nbsp; $M$&nbsp; symbols are equally probable.
+'''(4)'''&nbsp; The maximum entropy of a memoryless source with symbol set size&nbsp; $M$&nbsp; is obtained when all&nbsp; $M$&nbsp; symbols are equally probable.
-*For the special case $M=3$ it follows:
+*For the special case&nbsp; $M=3$&nbsp; it follows:
 :$$p_{\rm R} + p_{\rm G} + p_{\rm S} = 1 \hspace{0.3cm} \Rightarrow \hspace{0.3cm}
   \underline {p = 1/3 \approx 0.333}\hspace{0.05cm}.$$
@@ Line 110: / Line 110: @@
-'''(5)'''&nbsp; The system&nbsp; $\text{Roulette 1}$&nbsp; is information theoretically equal to the configuration&nbsp; $\rm Q_2$&nbsp; with&nbsp; $p = 1/37$:
+'''(5)'''&nbsp; The source model&nbsp; $\text{Roulette 1}$&nbsp; is information theoretically equal to the configuration&nbsp; $\rm Q_2$&nbsp; with&nbsp; $p = 1/37$:
 :$$p_{\rm G} = p =  \frac{1}{37}\hspace{0.05cm},\hspace{0.2cm} p_{\rm R} = p_{\rm S} = \frac{1-p}{2} = \frac{18}{37} \hspace{0.05cm}.$$
 *Thus, using the result of subtask&nbsp; '''(3)''', we obtain:
@@ Line 119: / Line 119: @@
-'''(6)'''&nbsp; If we bet on single numbers in roulette &nbsp;  &#8658; &nbsp; configuration&nbsp; $\text{Roulette 2}$, all numbers from&nbsp; $0$&nbsp; to&nbsp; $36$&nbsp;  are equally probable and we get:
+'''(6)'''&nbsp; If we bet on single numbers in roulette &nbsp;  &#8658; &nbsp; source model&nbsp; $\text{Roulette 2}$, all numbers from&nbsp; $0$&nbsp; to&nbsp; $36$&nbsp;  are equally probable and we get:
 :$$H = {\rm log}_2\hspace{0.1cm}(37)   \hspace{0.15cm} \underline { = 5.209 \, {\rm bit}}
   \hspace{0.05cm}.$$

@@ Line 4: / Line 4: @@
 [[File:Inf_A_1_2_vers2.png|right|frame|Probabilities of two ternary sources]]
-The entropy of a discrete-value memoryless message source with&nbsp; $M$&nbsp; possible symbols is:
+The entropy of a discrete-value memoryless source with&nbsp; $M$&nbsp; possible symbols is:
 :$$H =  \sum_{\mu = 1}^M p_{\mu} \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{p_\mu}\hspace{0.05cm},\hspace{0.3cm}
   {\rm pseudo unit\hspace{-0.15cm}: \hspace{0.15cm}bit}\hspace{0.05cm}.$$
-Here, the&nbsp; $p_\mu$&nbsp; denote the occurrence probabilities of the individual symbols or events.&nbsp; n the present example, the events are denoted by&nbsp; $\rm R$(ed),&nbsp; $\rm G$(reen)&nbsp; and&nbsp; $\rm S$(chwarz)&nbsp; with Schwarz being the German word for black.
+Here, the&nbsp; $p_\mu$&nbsp; denote the occurrence probabilities of the individual symbols or events.&nbsp; In the present example, the events are denoted by&nbsp; $\rm R$(ed),&nbsp; $\rm G$(reen)&nbsp; and&nbsp; $\rm S$(chwarz)&nbsp; with&nbsp; "Schwarz"&nbsp; being the German word for&nbsp; "Black".
 *For a binary source with the occurrence probabilities&nbsp; $p$ &nbsp;and&nbsp; $1-p$&nbsp; this can be written:
 :$$H = H_{\rm bin}(p) = p \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{p}+ (1-p) \cdot
 {\rm log}_2\hspace{0.1cm}\frac{1}{1-p}\hspace{0.05cm},\hspace{0.3cm}
-  {\rm pseudo unit\hspace{-0.15cm}: \hspace{0.15cm}bit}\hspace{0.05cm}.$$
+  \text{pseudo&ndash;unit: bit}\hspace{0.05cm}.$$
-*The entropy of a multilevel source can often be expressed with this&nbsp; "binary entropy function"&nbsp; $H_{\rm bin}(p)$&nbsp;.
+*The entropy of a multilevel source can often be expressed with this&nbsp; "binary entropy function"&nbsp; $H_{\rm bin}(p)$.
 In this task, two ternary sources with the symbol probabilities according to the above graph are considered:
-* source&nbsp; $\rm Q_1$ with&nbsp; $p_{\rm G }= 1/2$, &nbsp;$p_{\rm S }= 1/3$ &nbsp;and&nbsp; $p_{\rm R }= 1/6$,
+# source&nbsp; $\rm Q_1$ with&nbsp; $p_{\rm G }= 1/2$, &nbsp;$p_{\rm S }= 1/3$ &nbsp;and&nbsp; $p_{\rm R }= 1/6$,
-* source&nbsp; $\rm Q_2$ with&nbsp; $p_{\rm G }= p$ &nbsp;and&nbsp;  $p_{\rm S } = p_{\rm R } = (1-p)/2$.
+# source&nbsp; $\rm Q_2$ with&nbsp; $p_{\rm G }= p$ &nbsp;and&nbsp;  $p_{\rm S } = p_{\rm R } = (1-p)/2$.
-The ternary source&nbsp; $\rm Q_2$&nbsp; can also be applied to roulette when a player bets only on the squares&nbsp; $\rm R$,&nbsp; $\rm S$chwarz&nbsp; and $\rm G$reen&nbsp; (the "zero").&nbsp; This type of game is referred to as&nbsp; $\text{Roulette 1}$&nbsp; in the question section.
+*The ternary source&nbsp; $\rm Q_2$&nbsp; can also be applied to&nbsp; "Roulette"&nbsp; when a player bets only on the squares&nbsp; $\rm R$(ed),&nbsp; $\rm S$(chwarz)&nbsp; and $\rm G$(reen)&nbsp; (the "zero").&nbsp; This type of game is referred to as&nbsp; $\text{Roulette 1}$&nbsp; in the question section.
-In contrast,&nbsp; $\text{Roulette 2}$&nbsp; indicates that the player bets on single numbers&nbsp; $(0$, ... , $36)$&nbsp;.
+*In contrast,&nbsp; $\text{Roulette 2}$&nbsp; indicates that the player bets on single numbers&nbsp; $(0$, ... , $36)$.
@@ Line 46: / Line 46: @@
 {Which of the following statements are true if&nbsp;  $\rm R$,&nbsp; $\rm G$&nbsp; and&nbsp;  $\rm S$&nbsp; are represented by the numerical values&nbsp; $-1$, &nbsp;$0$ &nbsp;and&nbsp; $+1$&nbsp;?
-|type="[]"}
+|type="()"}
 - The result is a smaller entropy.
 + The entropy remains the same.
@@ Line 57: / Line 57: @@
-{For whic&nbsp; $p$&ndash;value  der Quelle&nbsp; $\rm \underline{Q_2}$&nbsp; does the maximum entropy result:&nbsp; $H &#8594; H_\text{max}$?
+{For which&nbsp; $p$&ndash;value of the source&nbsp; $\rm \underline{Q_2}$&nbsp; does the maximum entropy result:&nbsp; $H &#8594; H_\text{max}$?
 |type="{}"}
 $p \ =  \ $ { 0.333 3% }
-{What is the entropy of the message source&nbsp; $\text{Roulette 1}$,&nbsp; i.e. with respect to the events&nbsp; $\rm R$ed,&nbsp;  $\rm S$chwarz&nbsp; and&nbsp;  $\rm G$reen&nbsp; (the "zero")?
+{What is the entropy of the source model&nbsp; $\text{Roulette 1}$,&nbsp; i.e. with respect to the events&nbsp; $\rm R$(ed),&nbsp;  $\rm S$(chwarz)&nbsp; and&nbsp;  $\rm G$(reen)&nbsp; (the "zero")?
 |type="{}"}
 $H \ = \ $ { 1.152 3% } $\ \rm bit$

@@ Line 13: / Line 13: @@
 {\rm log}_2\hspace{0.1cm}\frac{1}{1-p}\hspace{0.05cm},\hspace{0.3cm}
   {\rm pseudo unit\hspace{-0.15cm}: \hspace{0.15cm}bit}\hspace{0.05cm}.$$
-*The entropy of a multilevel source can often be expressed with this&nbsp; &bdquo;binary entropy function&rdquo;&nbsp; $H_{\rm bin}(p)$&nbsp;.
+*The entropy of a multilevel source can often be expressed with this&nbsp; &bdquo;binary entropy function"&nbsp; $H_{\rm bin}(p)$&nbsp;.
@@ Line 22: / Line 22: @@
-The ternary source&nbsp; $\rm Q_2$&nbsp; can also be applied to roulette when a player bets only on the squares&nbsp; $\rm R$,&nbsp; $\rm S$chwarz&nbsp; and $\rm G$reen&nbsp; (the &bdquo;zero&rdquo;).&nbsp; This type of game is referred to as&nbsp; $\text{Roulette 1}$&nbsp; in the question section.
+The ternary source&nbsp; $\rm Q_2$&nbsp; can also be applied to roulette when a player bets only on the squares&nbsp; $\rm R$,&nbsp; $\rm S$chwarz&nbsp; and $\rm G$reen&nbsp; (the &bdquo;zero").&nbsp; This type of game is referred to as&nbsp; $\text{Roulette 1}$&nbsp; in the question section.
 In contrast,&nbsp; $\text{Roulette 2}$&nbsp; indicates that the player bets on single numbers&nbsp; $(0$, ... , $36)$&nbsp;.
@@ Line 62: / Line 62: @@
-{What is the entropy of the message source&nbsp; $\text{Roulette 1}$,&nbsp; i.e. with respect to the events&nbsp; $\rm R$ed,&nbsp;  $\rm S$chwarz&nbsp; and&nbsp;  $\rm G$reen&nbsp; (the &bdquo;zero&rdquo;)?
+{What is the entropy of the message source&nbsp; $\text{Roulette 1}$,&nbsp; i.e. with respect to the events&nbsp; $\rm R$ed,&nbsp;  $\rm S$chwarz&nbsp; and&nbsp;  $\rm G$reen&nbsp; (the &bdquo;zero")?
 |type="{}"}
 $H \ = \ $ { 1.152 3% } $\ \rm bit$

← Older revision		Revision as of 12:34, 13 May 2021
Line 126:		Line 126:


−	[[Category:Information Theory: Exercises\|^1.1 ~~Gedächtnislose Nachrichtenquellen~~^]]	+	[[Category:Information Theory: Exercises\|^1.1 Memoryless Sources^]]

@@ Line 77: / Line 77: @@
 ===Solution===
 {{ML-Kopf}}
-'''(1)'''&nbsp; Mit den Auftrittswahrscheinlichkeiten&nbsp; $1/2$,&nbsp; $1/3$&nbsp; und&nbsp; $1/6$&nbsp; erhält man folgenden Entropiewert:
+'''(1)'''&nbsp; With the probabilities of occurence&nbsp; $1/2$,&nbsp; $1/3$&nbsp; and&nbsp; $1/6$&nbsp; we get the following entropy value:
 :$$H \hspace{0.1cm}  =  \hspace{0.1cm}  1/2 \cdot {\rm log}_2\hspace{0.1cm}(2) +1/3 \cdot {\rm log}_2\hspace{0.1cm}(3) +1/6 \cdot {\rm log}_2\hspace{0.1cm}(6) =(1/2 + 1/6)\cdot {\rm log}_2\hspace{0.1cm}(2) +  (1/3 + 1/6)\cdot {\rm log}_2\hspace{0.1cm}(3) \hspace{0.15cm}\underline {\approx 1.46 \, {\rm bit}} \hspace{0.05cm}.$$
-'''(2)'''&nbsp; Richtig ist der <u>Lösungsvorschlag 2</u>:
+'''(2)'''&nbsp;<u>Proposed solution 2</u> is correct:
-* Die Entropie hängt nur von den Auftrittswahrscheinlichkeiten ab.
+*The entropy depends only on the probabilities of occurrence.
-*Es ist dabei egal, welche Zahlenwerte oder physikalische Größen man den einzelnen Symbolen zuordnet.
+*It does not matter which numerical values or physical quantities one assigns to the individual symbols.
-*Anders ist es bei Mittelwerten oder der AKF&ndash;Berechnung.&nbsp; Werden nur Symbole angegeben, so kann man hierfür keine Momente angeben.
+*It is different with mean values or the ACF (auto correlation function)calculation&nbsp; If only symbols are given, no moments can be given for them.
-*Außerdem hängen die Mittelwerte, Autokorrelation, usw. davon ab, ob man die Zuordnung bipolar&nbsp; $(-1, \hspace{0.10cm}0, \hspace{0.05cm}+1)$&nbsp; oder unipolar&nbsp;  $(0, \hspace{0.05cm}1, \hspace{0.05cm}2)$&nbsp; vereinbart.
+*Moreover, the mean values, autocorrelation, etc. depend on whether one agrees on the assignment bipolar&nbsp; $(-1, \hspace{0.10cm}0, \hspace{0.05cm}+1)$&nbsp;  or unipolar&nbsp;  $(0, \hspace{0.05cm}1, \hspace{0.05cm}2)$&nbsp;.
-'''(3)'''&nbsp; Die Entropie der Quelle&nbsp; $\rm Q_2$&nbsp; lässt sich wie folgt ausdrücken:
+'''(3)'''&nbsp; The entropy of source&nbsp; $\rm Q_2$&nbsp; can be expressed as follows:
 :$$H \hspace{0.1cm} =  \hspace{0.1cm} p \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p}+ 2 \cdot \frac{1-p}{2}  \cdot {\rm log}_2\hspace{0.1cm}\frac {2}{1-p}= p \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{p}+ (1-p)  \cdot {\rm log}_2\hspace{0.1cm}\frac {1}{1-p} + (1-p)\cdot {\rm log}_2\hspace{0.1cm}(2)= H_{\rm bin}(p) + 1-p \hspace{0.05cm}.$$
-*Für&nbsp; $p = 0.5$ &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H_{\rm bin}(p) = 1$&nbsp; ergibt sich&nbsp; $\underline{H = 1.5\hspace{0.05cm}\rm  bit}$.
+*For&nbsp; $p = 0.5$ &nbsp;&nbsp;&#8658;&nbsp;&nbsp; $H_{\rm bin}(p) = 1$&nbsp;, we get&nbsp; $\underline{H = 1.5\hspace{0.05cm}\rm  bit}$.
-'''(4)'''&nbsp; Die maximale Entropie einer gedächtnislosen Quelle mit dem Symbolumfang $M$ ergibt sich, wenn alle&nbsp; $M$&nbsp; Symbole gleichwahrscheinlich sind.
+'''(4)'''&nbsp; The maximum entropy of a memoryless source with symbol range $M$ is obtained when all&nbsp; $M$&nbsp; symbols are equally probable.
-*Für den Sonderfall $M=3$ folgt daraus:
+*For the special case $M=3$ it follows:
 :$$p_{\rm R} + p_{\rm G} + p_{\rm S} = 1 \hspace{0.3cm} \Rightarrow \hspace{0.3cm}
   \underline {p = 1/3 \approx 0.333}\hspace{0.05cm}.$$
-*Damit erhält man mit dem Ergebnis der Teilaufgabe&nbsp; '''(3)'''&nbsp; die folgende Entropie:
+*Thus, using the result of sub-task&nbsp; '''(3)'''&nbsp;, we obtain the following entropy:
 :$$H = H_{\rm bin}(1/3) + 1-1/3 = 1/3 \cdot
 {\rm log}_2\hspace{0.1cm}(3) + 2/3 \cdot {\rm log}_2\hspace{0.1cm}(3/2) + 2/3 $$
@@ Line 110: / Line 110: @@
-'''(5)'''&nbsp; Das System&nbsp; $\text{Roulette 1}$&nbsp; ist informationstheoretisch gleich der Konfiguration&nbsp; $\rm Q_2$&nbsp; mit&nbsp; $p = 1/37$:
+'''(5)'''&nbsp; The system&nbsp; $\text{Roulette 1}$&nbsp; is information theoretically equal to the configuration&nbsp; $\rm Q_2$&nbsp; with&nbsp; $p = 1/37$:
 :$$p_{\rm G} = p =  \frac{1}{37}\hspace{0.05cm},\hspace{0.2cm} p_{\rm R} = p_{\rm S} = \frac{1-p}{2} = \frac{18}{37} \hspace{0.05cm}.$$
-*Damit erhält man mit dem Ergebnis der Teilaufgabe&nbsp; '''(3)''':
+*Thus, using the result of subtask&nbsp; '''(3)''', we obtain:
 :$$H = H_{\rm bin}(1/37) + \frac{36}{37} = \frac{1}{37} \cdot {\rm log}_2\hspace{0.1cm}(37) + \frac{36}{37} \cdot {\rm log}_2\hspace{0.1cm}(37) - \frac{36}{37} \cdot {\rm log}_2\hspace{0.1cm}36 + \frac{36}{37} =
     {\rm log}_2\hspace{0.1cm}(37) + \frac{36}{37} \cdot ( 1- {\rm log}_2\hspace{0.1cm}(36)) = 5.209 - 4.057  \hspace{0.15cm} \underline { = 1.152 \, {\rm bit}}
@@ Line 119: / Line 119: @@
-'''(6)'''&nbsp; Setzt man bei Roulette auf einzelne Zahlen &nbsp;  &#8658; &nbsp; Konfiguration&nbsp; $\text{Roulette 2}$, so sind alle Zahlen von&nbsp; $0$&nbsp; bis&nbsp; $36$&nbsp; gleichwahrscheinlich und man erhält:
+'''(6)'''&nbsp; If we bet on single numbers in roulette &nbsp;  &#8658; &nbsp; configuration&nbsp; $\text{Roulette 2}$, all numbers from&nbsp; $0$&nbsp; to&nbsp; $36$&nbsp;  are equally probable and we get:
 :$$H = {\rm log}_2\hspace{0.1cm}(37)   \hspace{0.15cm} \underline { = 5.209 \, {\rm bit}}
   \hspace{0.05cm}.$$