Difference between revisions of "Information Theory"

Latest revision as of 18:50, 31 December 2023

Brief summary

From the earliest beginnings of message transmission as an engineering discipline, it has been the endeavour of many engineers and mathematicians to find a quantitative measure for the

contained $\rm information$ $($ quite generally: »the knowledge about something« $)$

in a $\rm message$ $($ here we mean »a collection of symbols and/or states» $)$ .

The $($ abstract $)$ information is communicated by the $($ concrete $)$ message and can be conceived as the interpretation of a message.

Claude Elwood Shannon succeeded in 1948, in establishing a consistent theory about the information content of messages, which was revolutionary in its time and created a new, still highly topical field of science: »Shannon's information theory« named after him.

This is what the fourth book in the $\rm LNTwww$ series deals with, in particular:

Entropy of discrete-value sources with and without memory, as well as natural message sources: Definition, meaning and computational possibilities.
Source coding and data compression, especially the »Lempel–Ziv–Welch method« and »Huffman's entropy encoding«.
Various entropies of two-dimensional discrete-value random quantities. Mutual information and channel capacity. Application to digital signal transmission.
Discrete-value information theory. Differential entropy. AWGN channel capacity with continuous-valued as well as discrete-valued input.

⇒ First a »content overview« on the basis of the »four main chapters« with a total of »13 individual chapters« and »106 sections«:

Exercises and multimedia

In addition to these theory pages, we also offer exercises and multimedia modules on this topic, which could help to clarify the teaching material:

$(1)$ $\text{Exercises}$

$(2)$ $\text{Learning videos}$

$(3)$ $\text{Applets}$

Further links

$(4)$ $\text{Bibliography}$

$(5)$ $\text{Impressum}$

@@ Line 1: / Line 1: @@
-Since the early beginnings of communications as an engineering discipline, many engineers and mathematicians have sought to find a quantitative measure of
+===Brief summary===
-*the  $\rm Information$ &nbsp; (in general: "the knowledge of something") contained in a&nbsp;  $\rm message$ &nbsp; (here we understand&nbsp; "a collection of symbols and/or states").
+{{BlueBox|TEXT=From the earliest beginnings of message transmission as an engineering discipline,&nbsp; it has been the endeavour of many engineers and mathematicians&nbsp; to find a quantitative measure for the
+*contained&nbsp;  $\rm information$ &nbsp;  $($ quite generally:&nbsp; &raquo;the knowledge about something&laquo; $)$
-The&nbsp; (abstract)&nbsp; information is communicated by the&nbsp; (concrete)&nbsp; message and can be seen as an interpretation of a message.
+*in a&nbsp;  $\rm message$ &nbsp;  $($ here we mean&nbsp; &raquo;a collection of symbols and/or states&raquo; $)$ .
-[https://de.wikipedia.org/wiki/Claude_Shannon Claude Elwood Shannon]&nbsp; succeeded in 1948 in establishing a consistent theory of the information content of messages,&nbsp; which was revolutionary in its time and created a new, still highly topical field of science:&nbsp; the theory named after him&nbsp;  $\text{Shannon's Information Theory}$ .
+The&nbsp;  $($ abstract $)$ &nbsp; information is communicated by the&nbsp;  $($ concrete $)$ &nbsp; message and can be conceived as the interpretation of a message.
-The subject matter corresponds to a&nbsp; $\text{lecture with two semester hours per week (SWS) and one additional SWS exercise}$.
+[https://en.wikipedia.org/wiki/Claude_Shannon '''Claude Elwood Shannon''']&nbsp; succeeded in 1948,&nbsp; in establishing a consistent theory about the information content of messages,&nbsp; which was revolutionary in its time and created a new,&nbsp; still highly topical field of science:&nbsp;  &raquo;'''Shannon's information theory&laquo;'''&nbsp; named after him.
-Here is a table of contents based on the&nbsp; $\text{four main chapters}$&nbsp; with a total of&nbsp; $\text{13 individual chapters}$.
+This is what the fourth book in the&nbsp;  $\rm LNTwww$  series deals with,&nbsp; in particular:
+# Entropy of discrete-value sources with and without memory,&nbsp; as well as natural message sources:&nbsp; Definition,&nbsp; meaning and computational possibilities.
+# Source coding and data compression,&nbsp; especially the &nbsp; &raquo;Lempel&ndash;Ziv&ndash;Welch method&laquo; &nbsp; and &nbsp; &raquo;Huffman's entropy encoding&laquo;.
+# Various entropies of two-dimensional discrete-value random quantities.&nbsp; Mutual information and channel capacity.&nbsp; Application to digital signal transmission.
+# Discrete-value information theory.&nbsp; Differential entropy.&nbsp; AWGN channel capacity with continuous-valued as well as discrete-valued input.
+&rArr; &nbsp; First a&nbsp; &raquo;'''content overview'''&laquo;&nbsp; on the basis of the&nbsp; &raquo;'''four main chapters'''&laquo;&nbsp; with a total of&nbsp; &raquo;'''13 individual chapters'''&laquo;&nbsp; and&nbsp; &raquo;'''106 sections'''&laquo;:}}
+===Content===
-===Contents===
 {{Collapsible-Kopf}}
 {{Collapse1| header=Entropy of Discrete Sources
@@ Line 37: / Line 47: @@
 |submenu=
 *[[/Differential Entropy/]]
-*[[/AWGN Channel Capacity for Continuous Input/]]
+*[[/AWGN Channel Capacity for Continuous-Valued Input/]]
-*[[/AWGN Channel Capacity for Discrete Input/]]
+*[[/AWGN Channel Capacity for Discrete-Valued Input/]]
 }}
 {{Collapsible-Fuß}}
-In addition to these theory pages, we also offer exercises and multimedia modules that could help to clarify the teaching material:
+===Exercises and multimedia===
-*[https://en.lntwww.de/Category:Information_Theory:_Exercises  $\text{Exercises}$ ]
+{{BlaueBox|TEXT=
-*[[LNTwww:Lernvideos_zu_Informationstheorie| $\text{Learning videos}$ ]]
+In addition to these theory pages,&nbsp; we also offer exercises and multimedia modules on this topic,&nbsp; which could help to clarify the teaching material:
-*[[LNTwww:HTML5-Applets_zu_Informationstheorie| $\text{Applets}$ ]]
-<br><br>
+ $(1)$ &nbsp; &nbsp; [https://en.lntwww.de/Category:Information_Theory:_Exercises  $\text{Exercises}$ ]
- $\text{More links:}$
-<br><br>
+$(2)$&nbsp; &nbsp; [[LNTwww:Learning_videos_to_"Information_Theory"|$\text{Learning videos}$]]
-<br><br>
- $\text{Other Links:}$
-<br><br>
-$(1)$&nbsp; &nbsp; [[LNTwww:Bibliography_to_Mobile_Communikations|$\text{Bibliography to the book}$]]
-$(2)$&nbsp; &nbsp; [[LNTwww:Notes_on_the_authors_and_materials_used_in the_preparation_of_Mobile Communications|$\text{Notes on the authors and materials used in the preparation of the book}$]]
+$(3)$&nbsp; &nbsp; [[LNTwww:Applets_to_"Information_Theory"|$\text{Applets}$]]&nbsp;}}
-<br><br>
+===Further links===
-$(1)$&nbsp; &nbsp; [[LNTwww:Literaturempfehlung_zu_Informationstheorie|$\text{Recommended literature for the book}$]]
+{{BlaueBox|TEXT=
+$(4)$&nbsp; &nbsp; [[LNTwww:Bibliography_to_"Information_Theory"|$\text{Bibliography}$]]
-$(2)$&nbsp; &nbsp; [[LNTwww:Weitere_Hinweise_zum_Buch_Informationstheorie|$\text{General notes about the book}$]] &nbsp; (Authors,&nbsp; other participants,&nbsp; materials as a starting point for the book,&nbsp; list of sources)
+$(5)$&nbsp; &nbsp; [[LNTwww:Imprint_for_the_book_"Information_Theory"|$\text{Impressum}$]]}}
 <br><br>
 __NOTOC__
 __NOEDITSECTION__

Difference between revisions of "Information Theory"

Latest revision as of 18:50, 31 December 2023

Brief summary

Content

Entropy of Discrete Sources

Source Coding - Data Compression

Mutual Information Between Two Discrete Random Variables

Information Theory for Continuous Random Variables

Exercises and multimedia

Further links