Information Theory/Natural Discrete Sources - Revision history

Guenter at 15:35, 14 February 2023

2023-02-14T15:35:55Z

Show changes

Guenter at 15:29, 14 February 2023

2023-02-14T15:29:05Z

Guenter at 15:28, 14 February 2023

2023-02-14T15:28:24Z

Hwang at 11:51, 13 February 2023

2023-02-13T11:51:23Z

Hwang at 11:50, 13 February 2023

2023-02-13T11:50:30Z

Hwang at 11:50, 13 February 2023

2023-02-13T11:50:03Z

Hwang at 11:47, 13 February 2023

2023-02-13T11:47:13Z

Hwang at 11:42, 13 February 2023

2023-02-13T11:42:18Z

Hwang at 11:40, 13 February 2023

2023-02-13T11:40:33Z

Hwang at 11:33, 13 February 2023

2023-02-13T11:33:29Z

@@ Line 45: / Line 45: @@
 :$$H_1 \approx 4.1\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
-'''(2)'''&nbsp; Regarding the&nbsp; &raquo;'''syllable frequency'''&laquo;&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://de.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding $\text{Friedrich Wilhelm Kaeding}$]&nbsp; '''KORREKTUR: link''' in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
+'''(2)'''&nbsp; Regarding the&nbsp; &raquo;'''syllable frequency'''&laquo;&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://de.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding $\text{Friedrich Wilhelm Kaeding}$]&nbsp;  in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
 :$$H_{\rm syllable} = \hspace{-0.1cm} H_{\rm root} + H_{\rm prefix} + H_{\rm final} + H_{\rm rest} \approx

@@ Line 45: / Line 45: @@
 :$$H_1 \approx 4.1\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
-'''(2)'''&nbsp; Regarding the&nbsp; &raquo;'''syllable frequency'''&laquo;&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://en.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding $\text{Friedrich Wilhelm Kaeding}$]&nbsp; '''KORREKTUR: link''' in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
+'''(2)'''&nbsp; Regarding the&nbsp; &raquo;'''syllable frequency'''&laquo;&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://de.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding $\text{Friedrich Wilhelm Kaeding}$]&nbsp; '''KORREKTUR: link''' in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
 :$$H_{\rm syllable} = \hspace{-0.1cm} H_{\rm root} + H_{\rm prefix} + H_{\rm final} + H_{\rm rest} \approx

@@ Line 61: / Line 61: @@
 :$$H_3 \approx {8.6}/{3.03}\approx 2.8\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
-'''(4)'''&nbsp; Küpfmueller's estimation of the entropy approximation&nbsp; $H_3$&nbsp; based mainly on the syllable frequencies according to&nbsp; '''(2)'''&nbsp; and the mean value of&nbsp; $3.03$&nbsp; letters per syllable. To get another entropy approximation&nbsp; $H_k$&nbsp; with greater&nbsp; $k$&nbsp; Küpfmüller additionally analyzed the words in German texts.&nbsp; He came to the following results:
+'''(4)'''&nbsp; Küpfmüller's estimation of the entropy approximation&nbsp; $H_3$&nbsp; based mainly on the syllable frequencies according to&nbsp; '''(2)'''&nbsp; and the mean value of&nbsp; $3.03$&nbsp; letters per syllable. To get another entropy approximation&nbsp; $H_k$&nbsp; with greater&nbsp; $k$&nbsp; Küpfmüller additionally analyzed the words in German texts.&nbsp; He came to the following results:
 :*The&nbsp; $322$&nbsp; most common words provide an entropy contribution of&nbsp; $4.5 \ \rm bit/word$.

@@ Line 40: / Line 40: @@
 :$$H_0 = \log_2 (26) ≈ 4.7\ \rm bit/letter.$$
-Küpfmueller's estimation is based on the following considerations:
+Küpfmüller's estimation is based on the following considerations:
 '''(1)'''&nbsp; The&nbsp; &raquo;'''first entropy approximation'''&laquo;&nbsp; results from the letter frequencies in German texts.&nbsp; According to a study of 1939, "e" is with a frequency of &nbsp; $16. 7\%$&nbsp; the most frequent, the rarest is "x" with&nbsp; $0.02\%$.&nbsp; Averaged over all letters we obtain&nbsp;

@@ Line 31: / Line 31: @@
 ==Entropy estimation according to Küpfmüller ==
 <br>
-Karl Küpfmüller has investigated the entropy of German texts in his published assessment &nbsp; [Küpf54]<ref name ='Küpf54'>Küpfmüller, K.:&nbsp; Die Entropie der deutschen Sprache.&nbsp; Fernmeldetechnische Zeitung 7, 1954, S. 265-272.</ref>&nbsp; the following assumptions are made:
+Karl Küpfmüller has investigated the entropy of German texts in his published assessment &nbsp; [Küpf54]<ref name ='Küpf54'>Küpfmüller, K.:&nbsp; Die Entropie der deutschen Sprache.&nbsp; Fernmeldetechnische Zeitung 7, 1954, S. 265-272.</ref>,&nbsp; the following assumptions are made:
 *an alphabet with&nbsp; $26$&nbsp; letters&nbsp; (no umlauts and punctuation marks),
 *not taking into account the space character,

← Older revision		Revision as of 11:47, 13 February 2023
Line 26:		Line 26:


−	A justified question is therefore:   How did  [https://en.wikipedia.org/wiki/Karl_K%C3%BCpfm%C3%BCller $\text{Karl Küpfmüller}$]  ~~determined~~ the entropy of the German language in 1954?  How did  [https://en.wikipedia.org/wiki/Claude_Shannon $\text{Claude Elwood Shannon}$]  do the same for the English language, even before Küpfmüller?  One thing is revealed beforehand:   Not with the approach described above.	+	A justified question is therefore:   How did  [https://en.wikipedia.org/wiki/Karl_K%C3%BCpfm%C3%BCller $\text{Karl Küpfmüller}$]  determine the entropy of the German language in 1954?  How did  [https://en.wikipedia.org/wiki/Claude_Shannon $\text{Claude Elwood Shannon}$]  do the same for the English language, even before Küpfmüller?  One thing is revealed beforehand:   Not with the approach described above.

@@ Line 19: / Line 19: @@
-Theoretically, one could also define the entropy here as the border crossing point of the entropy approximation&nbsp; $H_k$&nbsp; for&nbsp; $k \to \infty$,&nbsp; according to the procedure from the&nbsp; [[Information_Theory/Sources_with_Memory#Generalization to k -tuple and boundary crossing|"last chapter"]].&nbsp; In practice, however, insurmountable numerical limitations can be found here as well:
+Theoretically, one could also define the entropy here as the border crossing point of the entropy approximation&nbsp; $H_k$&nbsp; for&nbsp; $k \to \infty$,&nbsp; according to the procedure from the&nbsp; [[Information_Theory/Discrete_Sources_with_Memory#Generalization_to_.7F.27.22.60UNIQ-MathJax111-QINU.60.22.27.7F-tuple_and_boundary_crossing|"last chapter"]].&nbsp; In practice, however, insurmountable numerical limitations can be found here as well:
 *Already for the entropy approximation&nbsp; $H_2$&nbsp; there are&nbsp; $M^2 = 256^2 = 65\hspace{0.1cm}536$&nbsp; possible two-tuples.&nbsp; Thus, the calculation requires the same amount of memory (in bytes). &nbsp; If you assume that you need for a sufficiently safe statistic&nbsp; $100$&nbsp; equivalents per tuple on average,&nbsp; the length of the source symbol sequence should already be&nbsp; $N > 6.5 · 10^6$.
@@ Line 161: / Line 161: @@
-Further information on the synthetic generation of German and English texts can be found in the&nbsp; [[Aufgaben:1.8_Synthetisch_erzeugte_Texte|"Exercise 1.8"]].
+Further information on the synthetic generation of German and English texts can be found in the&nbsp; [[Aufgaben:Exercise_1.8:_Synthetically_Generated_Texts|"Exercise 1.8"]].
 ==Exercises for the chapter==
 <br>
-[[Aufgaben:1.7 Entropie natürlicher Texte|Aufgabe 1.7:  Entropie natürlicher Texte]]
+[[Aufgaben:Exercise_1.7:_Entropy_of_Natural_Texts|Exercise 1.7: Entropy of Natural Texts]]
-[[Aufgaben:1.8 Synthetisch erzeugte Texte|Aufgabe 1.8: Synthetisch erzeugte Texte]]
+[[Aufgaben:Exercise_1.8:_Synthetically_Generated_Texts|Exercise 1.8: Synthetically Generated Texts]]

@@ Line 26: / Line 26: @@
-A justified question is therefore: &nbsp; How did&nbsp; [https://en.wikipedia.org/wiki/Karl_K%C3%BCpfm%C3%BCller $\text{Karl Küpfmüller}$]&nbsp; determined the entropy of the German language in 1954?&nbsp; How did&nbsp; [https://en.wikipedia.org/wiki/Claude_Shannon Claude Elwood Shannon]&nbsp; do the same for the English language, even before Küpfmüller?&nbsp; One thing is revealed beforehand: &nbsp; Not with the approach described above.
+A justified question is therefore: &nbsp; How did&nbsp; [https://en.wikipedia.org/wiki/Karl_K%C3%BCpfm%C3%BCller $\text{Karl Küpfmüller}$]&nbsp; determined the entropy of the German language in 1954?&nbsp; How did&nbsp; [https://en.wikipedia.org/wiki/Claude_Shannon $\text{Claude Elwood Shannon}$]&nbsp; do the same for the English language, even before Küpfmüller?&nbsp; One thing is revealed beforehand: &nbsp; Not with the approach described above.
@@ Line 42: / Line 42: @@
 Küpfmueller's estimation is based on the following considerations:
-'''(1)'''&nbsp; The&nbsp; '''first entropy approximation'''&nbsp; results from the letter frequencies in German texts.&nbsp; According to a study of 1939, "e" is with a frequency of &nbsp; $16. 7\%$&nbsp; the most frequent, the rarest is "x" with&nbsp; $0.02\%$.&nbsp; Averaged over all letters we obtain&nbsp;
+'''(1)'''&nbsp; The&nbsp; &raquo;'''first entropy approximation'''&laquo;&nbsp; results from the letter frequencies in German texts.&nbsp; According to a study of 1939, "e" is with a frequency of &nbsp; $16. 7\%$&nbsp; the most frequent, the rarest is "x" with&nbsp; $0.02\%$.&nbsp; Averaged over all letters we obtain&nbsp;
 :$$H_1 \approx 4.1\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
-'''(2)'''&nbsp; Regarding the&nbsp; '''syllable frequency'''&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://en.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding Friedrich Wilhelm Kaeding]&nbsp; in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
+'''(2)'''&nbsp; Regarding the&nbsp; &raquo;'''syllable frequency'''&laquo;&nbsp; Küpfmüller evaluates the&nbsp; "Häufigkeitswörterbuch der deutschen Sprache"&nbsp; (Frequency Dictionary of the German Language), published by&nbsp; [https://en.wikipedia.org/wiki/Friedrich_Wilhelm_Kaeding $\text{Friedrich Wilhelm Kaeding}$]&nbsp; in 1898.&nbsp; He distinguishes between root syllables, prefixes, and final syllables and thus arrives at the average information content of all syllables:
 :$$H_{\rm syllable} = \hspace{-0.1cm} H_{\rm root} + H_{\rm prefix} + H_{\rm final} + H_{\rm rest} \approx
@@ Line 58: / Line 58: @@
-'''(3)'''&nbsp; As average number of letters per syllable Küpfmüller determined the value&nbsp; $3.03$.&nbsp; From this he deduced the&nbsp; '''third entropy approximation''''&nbsp; regarding the letters:
+'''(3)'''&nbsp; As average number of letters per syllable Küpfmüller determined the value&nbsp; $3.03$.&nbsp; From this he deduced the&nbsp; &raquo;'''third entropy approximation'''&laquo;&nbsp; regarding the letters:
 :$$H_3 \approx {8.6}/{3.03}\approx 2.8\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
@@ Line 64: / Line 64: @@
 :*The&nbsp; $322$&nbsp; most common words provide an entropy contribution of&nbsp; $4.5 \ \rm bit/word$.
-:*The contributions of the remaining&nbsp; $40\hspace{0.1cm}000$ words&nbsp; were estimated.&nbsp; Assuming that the frequencies of rare words are reciprocal to their ordinal number ([https://en.wikipedia.org/wiki/Zipf%27s_law Zipf's Law]).
+:*The contributions of the remaining&nbsp; $40\hspace{0.1cm}000$ words&nbsp; were estimated.&nbsp; Assuming that the frequencies of rare words are reciprocal to their ordinal number ([https://en.wikipedia.org/wiki/Zipf%27s_law $\text{Zipf's Law}$]).
 :*With these assumptions the average information content (related to words) is about &nbsp; $11 \ \rm bit/word$.
@@ Line 70: / Line 70: @@
 '''(5)'''&nbsp; The counting "letters per word" resulted in average&nbsp; $5.5$.&nbsp; Analogous to point&nbsp; '''(3)'''&nbsp; the entropy approximation for&nbsp; $k = 5.5$&nbsp; was approximated.&nbsp; Küpfmüller gives the value:&nbsp;
 :$$H_{5.5} \approx {11}/{5.5}\approx 2\,\, {\rm bit/letter}\hspace{0.05 cm}.$$
-:Of course,&nbsp; $k$&nbsp; can only assume integer values,&nbsp; according to&nbsp; [[Information_Theory/Sources_with_Memory#Generalization to k-tuple and boundary crossing|its definition]].&nbsp; This equation is therefore to be interpreted in such a way that for&nbsp; $H_5$&nbsp; a somewhat larger and for&nbsp; $H_6$&nbsp; a somewhat smaller value than&nbsp; $2 \ {\rm bit/letter}$&nbsp; will result.
+:Of course,&nbsp; $k$&nbsp; can only assume integer values,&nbsp; according to&nbsp; [[Information_Theory/Sources_with_Memory#Generalization to k-tuple and boundary crossing|$\text{its definition}$]].&nbsp; This equation is therefore to be interpreted in such a way that for&nbsp; $H_5$&nbsp; a somewhat larger and for&nbsp; $H_6$&nbsp; a somewhat smaller value than&nbsp; $2 \ {\rm bit/letter}$&nbsp; will result.
@@ Line 161: / Line 161: @@
-Further information on the synthetic generation of German and English texts can be found in the&nbsp; [[Aufgaben:1.8_Synthetisch_erzeugte_Texte|Exercise 1.8]].
+Further information on the synthetic generation of German and English texts can be found in the&nbsp; [[Aufgaben:1.8_Synthetisch_erzeugte_Texte|"Exercise 1.8"]].