Information Theory

Since the early beginnings of communications as an engineering discipline,  many engineers and mathematicians have sought

• to find a quantitative measure of the  $\rm information$  $($in general: "the knowledge of something"$)$

• contained in a  $\rm message$  $($here we understand  "a collection of symbols and/or states"$)$.

The  (abstract)  information is communicated by the  (concrete)  message and can be seen as an interpretation of a message.

Claude Elwood Shannon  succeeded in 1948 in establishing a consistent theory of the information content of messages, 
which was revolutionary in its time and created a new,  still highly topical field of science: 
The theory named after him:  $\text{Shannon's Information Theory}$.

Here first a  »content overview«  on the basis of the  »four main chapters«  with a total of  »13 individual chapters«:

Contents

Entropy of Discrete Sources

• Discrete Memoryless Sources
• Discrete Sources with Memory
• Natural Discrete Sources

Source Coding - Data Compression

• General Description
• Compression According to Lempel, Ziv and Welch
• Entropy Coding According to Huffman
• Further Source Coding Methods

Mutual Information Between Two Discrete Random Variables

• Some Preliminary Remarks on Two-Dimensional Random Variables
• Different Entropy Measures of Two-Dimensional Random Variables
• Application to Digital Signal Transmission

Information Theory for Continuous Random Variables

• Differential Entropy
• AWGN Channel Capacity for Continuous-Valued Input
• AWGN Channel Capacity for Discrete-Valued Input

In addition to these theory pages, we also offer exercises and multimedia modules that could help to clarify the teaching material:

• $\text{Exercises}$
• $\text{Learning videos}$
• $\text{Applets}$

$\text{Other links:}$

$(1)$    $\text{Bibliography to the book}$

$(2)$    $\text{General notes about the book}$   (authors,  other participants,  materials as a starting point for the book,  list of sources)