Difference between revisions of "Information Theory"

Latest revision as of 18:50, 31 December 2023

Brief summary

From the earliest beginnings of message transmission as an engineering discipline, it has been the endeavour of many engineers and mathematicians to find a quantitative measure for the

contained $\rm information$ $($ quite generally: »the knowledge about something« $)$

in a $\rm message$ $($ here we mean »a collection of symbols and/or states» $)$ .

The $($ abstract $)$ information is communicated by the $($ concrete $)$ message and can be conceived as the interpretation of a message.

Claude Elwood Shannon succeeded in 1948, in establishing a consistent theory about the information content of messages, which was revolutionary in its time and created a new, still highly topical field of science: »Shannon's information theory« named after him.

This is what the fourth book in the $\rm LNTwww$ series deals with, in particular:

Entropy of discrete-value sources with and without memory, as well as natural message sources: Definition, meaning and computational possibilities.
Source coding and data compression, especially the »Lempel–Ziv–Welch method« and »Huffman's entropy encoding«.
Various entropies of two-dimensional discrete-value random quantities. Mutual information and channel capacity. Application to digital signal transmission.
Discrete-value information theory. Differential entropy. AWGN channel capacity with continuous-valued as well as discrete-valued input.

⇒ First a »content overview« on the basis of the »four main chapters« with a total of »13 individual chapters« and »106 sections«:

Exercises and multimedia

In addition to these theory pages, we also offer exercises and multimedia modules on this topic, which could help to clarify the teaching material:

$(1)$ $\text{Exercises}$

$(2)$ $\text{Learning videos}$

$(3)$ $\text{Applets}$

Further links

$(4)$ $\text{Bibliography}$

$(5)$ $\text{Impressum}$

@@ Line 1: / Line 1: @@
-Since the early beginnings of communications as an engineering discipline, many engineers and mathematicians have sought to find a quantitative measure of
+===Brief summary===
-*the  $\rm Information$ &nbsp; (in general: "the knowledge of something") contained in a&nbsp;  $\rm message$ &nbsp; (here we understand "a collection of symbols and/or states").
+{{BlueBox|TEXT=From the earliest beginnings of message transmission as an engineering discipline,&nbsp; it has been the endeavour of many engineers and mathematicians&nbsp; to find a quantitative measure for the
+*contained&nbsp;  $\rm information$ &nbsp;  $($ quite generally:&nbsp; &raquo;the knowledge about something&laquo; $)$
-The (abstract) information is communicated by the (concrete) message and can be seen as an interpretation of a message.
+*in a&nbsp;  $\rm message$ &nbsp; $( $here we mean&nbsp; &raquo;a collection of symbols and/or states&raquo;$ )$.
-[https://de.wikipedia.org/wiki/Claude_Shannon Claude Elwood Shannon]&nbsp; succeeded in 1948 in establishing a consistent theory of the information content of messages, which was revolutionary in its time and created a new, still highly topical field of science:&nbsp; the theory named after him&nbsp; $\text{Shannon's Information Theory}$.
+The&nbsp;  $($ abstract $)$ &nbsp; information is communicated by the&nbsp; $( $concrete$ )$&nbsp; message and can be conceived as the interpretation of a message.
-The course material corresponds to a&nbsp; $\text{lecture with two semester hours per week (SWS) and one SWS exercise}$.
+[https://en.wikipedia.org/wiki/Claude_Shannon '''Claude Elwood Shannon''']&nbsp; succeeded in 1948,&nbsp; in establishing a consistent theory about the information content of messages,&nbsp; which was revolutionary in its time and created a new,&nbsp; still highly topical field of science:&nbsp;  &raquo;'''Shannon's information theory&laquo;'''&nbsp; named after him.
-Here is a table of contents based on the&nbsp; $\text{four main chapters}$&nbsp; with a total of&nbsp; $\text{13 individual chapters}$.
+This is what the fourth book in the&nbsp;  $\rm LNTwww$  series deals with,&nbsp; in particular:
+# Entropy of discrete-value sources with and without memory,&nbsp; as well as natural message sources:&nbsp; Definition,&nbsp; meaning and computational possibilities.
+# Source coding and data compression,&nbsp; especially the &nbsp; &raquo;Lempel&ndash;Ziv&ndash;Welch method&laquo; &nbsp; and &nbsp; &raquo;Huffman's entropy encoding&laquo;.
+# Various entropies of two-dimensional discrete-value random quantities.&nbsp; Mutual information and channel capacity.&nbsp; Application to digital signal transmission.
+# Discrete-value information theory.&nbsp; Differential entropy.&nbsp; AWGN channel capacity with continuous-valued as well as discrete-valued input.
+&rArr; &nbsp; First a&nbsp; &raquo;'''content overview'''&laquo;&nbsp; on the basis of the&nbsp; &raquo;'''four main chapters'''&laquo;&nbsp; with a total of&nbsp; &raquo;'''13 individual chapters'''&laquo;&nbsp; and&nbsp; &raquo;'''106 sections'''&laquo;:}}
+===Content===
-===Inhalt===
 {{Collapsible-Kopf}}
 {{Collapse1| header=Entropy of Discrete Sources
 | submenu=
 *[[/Discrete Memoryless Sources/]]
-*[[/Sources with Memory/]]
+*[[/Discrete Sources with Memory/]]
 *[[/Natural Discrete Sources/]]
 }}
@@ Line 30: / Line 41: @@
 |submenu=
 *[[/Some Preliminary Remarks on Two-Dimensional Random Variables/]]
-*[[/Different Entropies of Two-Dimensional Random Variables/]]
+*[[/Different Entropy Measures of Two-Dimensional Random Variables/]]
 *[[/Application to Digital Signal Transmission/]]
 }}
 {{Collapse4 | header=Information Theory for Continuous Random Variables
 |submenu=
-*[[/Differential entropy/]]
+*[[/Differential Entropy/]]
-*[[/AWGN channel capacity with continuous value input/]]
+*[[/AWGN Channel Capacity for Continuous-Valued Input/]]
-*[[/AWGN channel capacity with discrete value input/]]
+*[[/AWGN Channel Capacity for Discrete-Valued Input/]]
 }}
 {{Collapsible-Fuß}}
-In addition to these theory pages, we also offer Exercises and multimedia modules that could help to clarify the teaching material:
+===Exercises and multimedia===
-*[https://en.lntwww.de/Kategorie:Aufgaben_zu_Informationstheorie  $\text{Exercises}$ ]
-*[[LNTwww:Lernvideos_zu_Informationstheorie| $\text{Learning videos}$ ]]
+{{BlaueBox|TEXT=
-*[[LNTwww:HTML5-Applets_zu_Informationstheorie| $\text{redesigned applets}$ ]], based on HTML5, also executable on smartphones
+In addition to these theory pages,&nbsp; we also offer exercises and multimedia modules on this topic,&nbsp; which could help to clarify the teaching material:
-*[[LNTwww:SWF-Applets_zu_Informationstheorie| $\text{former Applets}$ ]], based on SWF, executable only under WINDOWS with ''Adobe Flash Player''.
+ $(1)$ &nbsp; &nbsp; [https://en.lntwww.de/Category:Information_Theory:_Exercises  $\text{Exercises}$ ]
+ $(2)$ &nbsp; &nbsp; [[LNTwww:Learning_videos_to_"Information_Theory"| $\text{Learning videos}$ ]]
+ $(3)$ &nbsp; &nbsp; [[LNTwww:Applets_to_"Information_Theory"| $\text{Applets}$ ]]&nbsp;}}
+===Further links===
+{{BlaueBox|TEXT=
+ $(4)$ &nbsp; &nbsp; [[LNTwww:Bibliography_to_"Information_Theory"| $\text{Bibliography}$ ]]
+ $(5)$ &nbsp; &nbsp; [[LNTwww:Imprint_for_the_book_"Information_Theory"| $\text{Impressum}$ ]]}}
 <br><br>
- $\text{More links:}$
-<br><br>
- $(1)$ &nbsp; &nbsp; [[LNTwww:Literaturempfehlung_zu_Informationstheorie| $\text{Recommended literature for the book}$ ]]
- $(2)$ &nbsp; &nbsp; [[LNTwww:Weitere_Hinweise_zum_Buch_Informationstheorie| $\text{General notes about the book}$ ]] &nbsp; (Authors,&nbsp; other participants,&nbsp; materials as a starting point for the book,&nbsp; list of sources)
-<br><br>
 __NOTOC__
 __NOEDITSECTION__

Difference between revisions of "Information Theory"

Latest revision as of 18:50, 31 December 2023

Brief summary

Content

Entropy of Discrete Sources

Source Coding - Data Compression

Mutual Information Between Two Discrete Random Variables

Information Theory for Continuous Random Variables

Exercises and multimedia

Further links