View source for Applets:Capacity of Memoryless Digital Channels

{{LntAppletLinkEnDe|transinformation_en|transinformation}}
 

==Applet Description==
<br>
In this applet, binary&nbsp; $(M=2)$&nbsp; and ternary&nbsp; $(M=3)$&nbsp; channel models without memory are considered with&nbsp; $M$&nbsp; inputs&nbsp; $X$&nbsp; and&nbsp; $M$&nbsp; outputs&nbsp; $Y$.&nbsp; Such a message system is completely determined by the probability function&nbsp; $P_X(X)$&nbsp; and the matrix&nbsp; $P_{\hspace{0.01cm}Y\hspace{0.03cm} \vert \hspace{0.01cm}X}(Y\hspace{0.03cm}  \vert \hspace{0.03cm} X)$&nbsp; of the transition probabilities.


For these binary and ternary systems, the following information-theoretic descriptive quantities are derived and clarified:
*the&nbsp; ''source entropy'' &nbsp; $H(X)$&nbsp; and the&nbsp; ''sink entropy'' &nbsp; $H(Y)$,
*the&nbsp; ''equivocation'' &nbsp; $H(X|Y)$&nbsp; and the &nbsp; ''irrelevance'' &nbsp; $H(Y|X)$,
*the&nbsp; ''joint entropy'' &nbsp; $H(XY)$&nbsp; as well as the ''mutual information''&nbsp;  $I(X; Y)$,
*the&nbsp; ''channel capacity'' &nbsp; as the decisive parameter of digital channel models without memory:
:$$C = \max_{P_X(X)} \hspace{0.15cm}  I(X;Y)  \hspace{0.05cm}.$$ 

These information-theoretical quantities can be calculated both in analytic&ndash;closed form or determined simulatively by evaluation of source and sink symbol sequence.

==Theoretical Background==
<br>
===Underlying model of digital signal transmission === 	

The set of possible&nbsp; &raquo;'''source symbols'''&laquo;&nbsp; is characterized by the discrete random variable&nbsp; $X$.&nbsp; 
*In the binary case &nbsp; &rArr; &nbsp; $M_X= |X| = 2$&nbsp; holds&nbsp; $X = \{\hspace{0.05cm}{\rm A}, \hspace{0.15cm} {\rm B} \hspace{0.05cm}\}$&nbsp; with the probability mass function &nbsp; $($ $\rm PMF)$  &nbsp; $P_X(X)= \big (p_{\rm A},\hspace{0.15cm}p_{\rm B}\big)$&nbsp; and the source symbol probabilities&nbsp; $p_{\rm A}$&nbsp; and&nbsp; $p_{\rm B}=1- p_{\rm A}$.
*Accordingly, for a ternary source&nbsp; &rArr; &nbsp; $M_X= |X| = 3$: &nbsp; &nbsp; $X = \{\hspace{0.05cm}{\rm A}, \hspace{0.15cm} {\rm B}, \hspace{0.15cm} {\rm C} \hspace{0.05cm}\}$, &nbsp; &nbsp; $P_X(X)= \big (p_{\rm A},\hspace{0.15cm}p_{\rm B},\hspace{0.15cm}p_{\rm C}\big)$, &nbsp; &nbsp; $p_{\rm C}=1- p_{\rm A}-p_{\rm B}$.


The set of possible&nbsp; &raquo;'''sink symbols'''&laquo;&nbsp; is characterized by the discrete random variable&nbsp; $Y$.&nbsp; These come from the same set of symbols as the source symbols &nbsp; &rArr; &nbsp; $M_Y=M_X = M$.&nbsp; To simplify the following description, we denote them with lowercase letters, for example, for&nbsp; $M=3$: &nbsp;&nbsp; $Y = \{\hspace{0.05cm}{\rm a}, \hspace{0.15cm} {\rm b}, \hspace{0.15cm} {\rm c} \hspace{0.05cm}\}$.   

The relationship between the random variables&nbsp; $X$&nbsp; and&nbsp; $Y$&nbsp; is given by a &nbsp; &raquo;'''discrete memoryless channel model'''&laquo;&nbsp; $($ $\rm DMC)$.&nbsp; The left graph shows this for&nbsp; $M=2$&nbsp; and the right graph for&nbsp; $M=3$.

[[File:Transinf_1_neu.png|center|frame|&nbsp; $M=2$&nbsp; (left) and for&nbsp; $M=3$&nbsp; (right). <br>Please note:&nbsp; In the right graph not all transitions are labeled]]

The following description applies to the simpler case&nbsp; $M=2$.&nbsp; For the calculation of all information theoretic quantities in the next section we need, besides&nbsp; $P_X(X)$&nbsp; and&nbsp;  $P_Y(Y)$&nbsp; the two-dimensional probability functions&nbsp; $($each a&nbsp; $2\times2$&ndash;matrix$)$&nbsp; of all
#&nbsp; [[Theory_of_Stochastic_Signals/Statistical_Dependence_and_Independence#Conditional_probability|$\text{conditional probabilities}$]] &nbsp; &rArr; &nbsp; $P_{\hspace{0.01cm}Y\hspace{0.03cm} \vert \hspace{0.01cm}X}(Y\hspace{0.03cm}  \vert \hspace{0.03cm} X)$ &nbsp; &rArr; &nbsp; given by the DMC model;
#&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Joint_probability_and_joint_entropy|$\text{joint probabilities}$]]&nbsp; &rArr; &nbsp; $P_{XY}(X,\hspace{0.1cm}Y)$;
#&nbsp; [[Theory_of_Stochastic_Signals/Statistical_Dependence_and_Independence#Inference_probability|$\text{inference probabilities}$]] &nbsp; &rArr; &nbsp; $P_{\hspace{0.01cm}X\hspace{0.03cm} \vert \hspace{0.03cm}Y}(X\hspace{0.03cm}  \vert \hspace{0.03cm} Y)$.
 

[[File:Transinf_2.png|right|frame|Considered model of the binary channel]]
{{GraueBox|TEXT=
$\text{Example 1}$:&nbsp; We consider the sketched binary channel.
* Let the falsification probabilities be:
  
:$$\begin{align*}p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}A}  & =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm a}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm A}) = 0.95\hspace{0.05cm},\hspace{0.8cm}p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}A}  =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm b}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm A}) = 0.05\hspace{0.05cm},\\
p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}B}  & =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm a}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm B}) = 0.40\hspace{0.05cm},\hspace{0.8cm}p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}B}  =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm b}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm B}) = 0.60\end{align*}$$

:$$\Rightarrow \hspace{0.3cm}  P_{\hspace{0.01cm}Y\hspace{0.05cm} \vert \hspace{0.05cm}X}(Y\hspace{0.05cm}  \vert \hspace{0.05cm} X) = 
		\begin{pmatrix} 
		0.95  & 0.05\\
		0.4 & 0.6
		\end{pmatrix} \hspace{0.05cm}.$$

*Furthermore, we assume source symbols that are not equally probable:
 
:$$P_X(X) = \big ( p_{\rm A},\ p_{\rm B} \big )=
\big ( 0.1,\ 0.9 \big )
\hspace{0.05cm}.$$

*Thus, for the probability function of the sink we get:
   
:$$P_Y(Y) = \big [ {\rm Pr}( Y\hspace{-0.1cm} = {\rm a})\hspace{0.05cm}, \ {\rm Pr}( Y \hspace{-0.1cm}= {\rm b}) \big ] = \big ( 0.1\hspace{0.05cm},\ 0.9 \big ) \cdot 
\begin{pmatrix} 
		0.95  & 0.05\\
		0.4 & 0.6
		\end{pmatrix} $$

:$$\Rightarrow \hspace{0.3cm}  {\rm Pr}( Y \hspace{-0.1cm}= {\rm a}) =  
0.1 \cdot 0.95 + 0.9 \cdot 0.4 = 0.455\hspace{0.05cm},\hspace{1.0cm}
{\rm Pr}( Y \hspace{-0.1cm}= {\rm b})  =  1 - {\rm Pr}( Y \hspace{-0.1cm}= {\rm a}) = 0.545.$$

*The joint probabilities&nbsp; $p_{\mu \kappa} = \text{Pr}\big[(X = μ) ∩ (Y = κ)\big]$&nbsp; between source and sink are:
 
:$$\begin{align*}p_{\rm Aa} & =   p_{\rm a} \cdot p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}A} = 0.095\hspace{0.05cm},\hspace{0.5cm}p_{\rm Ab} =   p_{\rm b} \cdot p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}A} = 0.005\hspace{0.05cm},\\
p_{\rm Ba} & =   p_{\rm a} \cdot p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}B} = 0.360\hspace{0.05cm},
\hspace{0.5cm}p_{\rm Bb} =   p_{\rm b} \cdot p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}B} = 0.540\hspace{0.05cm}.
\end{align*}$$

:$$\Rightarrow \hspace{0.3cm}  P_{XY}(X,\hspace{0.1cm}Y) = 
		\begin{pmatrix} 
		0.095  & 0.005\\
		0.36 & 0.54
		\end{pmatrix} \hspace{0.05cm}.$$
 
* 

:$$\begin{align*}p_{\rm A\hspace{0.03cm}\vert \hspace{0.03cm}a} & =   p_{\rm Aa}/p_{\rm a} = 0.095/0.455 = 0.2088\hspace{0.05cm},\hspace{0.5cm}p_{\rm A\hspace{0.03cm}\vert \hspace{0.03cm}b}  =   p_{\rm Ab}/p_{\rm b} = 0.005/0.545 = 0.0092\hspace{0.05cm},\\
p_{\rm B\hspace{0.03cm}\vert \hspace{0.03cm}a} & =   p_{\rm Ba}/p_{\rm a} = 0.36/0.455 = 0.7912\hspace{0.05cm},\hspace{0.5cm}p_{\rm B\hspace{0.03cm}\vert \hspace{0.03cm}b}  =   p_{\rm Bb}/p_{\rm b} = 0.54/0.545 = 0.9908\hspace{0.05cm}
\end{align*}$$

:$$\Rightarrow \hspace{0.3cm}  P_{\hspace{0.01cm}X\hspace{0.05cm} \vert \hspace{0.05cm}Y}(X\hspace{0.05cm}  \vert \hspace{0.05cm} Y) = 
		\begin{pmatrix} 
		0.2088  & 0.0092\\
		0.7912 & 0.9908
		\end{pmatrix} \hspace{0.05cm}.$$ }}
<br clear=all><br><br>
===Definition and interpretation of various entropy functions ===

In the&nbsp; [[Information_Theory/Verschiedene_Entropien_zweidimensionaler_Zufallsgrößen|$\text{$\rm LNTwww$ theory section}$]],&nbsp; all entropies relevant for 2D random quantities are defined, which also apply to digital signal transmission.&nbsp; In addition, you will find two diagrams illustrating the relationship between the individual entropies.&nbsp; 
*For digital signal transmission the right representation is appropriate, where the direction from source&nbsp; $X$&nbsp; to the sink&nbsp; $Y$&nbsp; is recognizable.&nbsp; 
*We now interpret the individual information-theoretical quantities on the basis of this diagram.


[[File:P_ID2781__Inf_T_3_3_S2.png|center|frame|Two information-theoretic models for digital signal transmission.
<br>Please note:&nbsp; In the right graph&nbsp; $H_{XY}$&nbsp; cannot be represented]]

*The&nbsp; &raquo;'''source entropy'''&laquo;&nbsp; $H(X)$&nbsp; denotes the average information content of the source symbol sequence.&nbsp; With the symbol set size&nbsp; $|X|$&nbsp; applies:
 
:$$H(X) = {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_X(X)}\right ] \hspace{0.1cm}
= -{\rm E} \big [ {\rm log}_2 \hspace{0.1cm}{P_X(X)}\big ] \hspace{0.2cm}
=\hspace{0.2cm} \sum_{\mu = 1}^{|X|} 
 P_X(x_{\mu}) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{P_X(x_{\mu})} \hspace{0.05cm}.$$

*The&nbsp; &raquo;'''equivocation'''&laquo;&nbsp; $H(X|Y)$&nbsp;  indicates the average information content that an observer who knows exactly about the sink&nbsp; $Y$&nbsp; gains by observing the source&nbsp; $X$&nbsp;:
 
:$$H(X|Y) = {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{\hspace{0.05cm}X\hspace{-0.01cm}|\hspace{-0.01cm}Y}(X\hspace{-0.01cm} |\hspace{0.03cm} Y)}\right ] \hspace{0.2cm}=\hspace{0.2cm} \sum_{\mu = 1}^{|X|} \sum_{\kappa = 1}^{|Y|} 
 P_{XY}(x_{\mu},\hspace{0.05cm}y_{\kappa}) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{\hspace{0.05cm}X\hspace{-0.01cm}|\hspace{0.03cm}Y}
(\hspace{0.05cm}x_{\mu}\hspace{0.03cm} |\hspace{0.05cm} y_{\kappa})}
 \hspace{0.05cm}.$$

*The equivocation is the portion of the source entropy&nbsp; $H(X)$&nbsp; that is lost due to channel interference&nbsp; (for digital channel: transmission errors).&nbsp; The&nbsp; &raquo;'''mutual information'''&laquo;&nbsp; $I(X; Y)$&nbsp; remains, which reaches the sink:
 
:$$I(X;Y) = {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{P_{XY}(X, Y)}{P_X(X) \cdot P_Y(Y)}\right ] \hspace{0.2cm}=\hspace{0.2cm} \sum_{\mu = 1}^{|X|} \sum_{\kappa = 1}^{|Y|} 
 P_{XY}(x_{\mu},\hspace{0.05cm}y_{\kappa}) \cdot {\rm log}_2 \hspace{0.1cm} \frac{P_{XY}(x_{\mu},\hspace{0.05cm}y_{\kappa})}{P_{\hspace{0.05cm}X}(\hspace{0.05cm}x_{\mu}) \cdot P_{\hspace{0.05cm}Y}(\hspace{0.05cm}y_{\kappa})}
 \hspace{0.05cm} = H(X) - H(X|Y) \hspace{0.05cm}.$$

*The&nbsp; &raquo;'''irrelevance'''&laquo;&nbsp; $H(Y|X)$&nbsp; indicates the average information content that an observer who knows exactly about the source&nbsp; $X$&nbsp; gains by observing the sink&nbsp; $Y$:
 
:$$H(Y|X) = {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{\hspace{0.05cm}Y\hspace{-0.01cm}|\hspace{-0.01cm}X}(Y\hspace{-0.01cm} |\hspace{0.03cm} X)}\right ] \hspace{0.2cm}=\hspace{0.2cm} \sum_{\mu = 1}^{|X|} \sum_{\kappa = 1}^{|Y|} 
 P_{XY}(x_{\mu},\hspace{0.05cm}y_{\kappa}) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{\hspace{0.05cm}Y\hspace{-0.01cm}|\hspace{0.03cm}X}
(\hspace{0.05cm}y_{\kappa}\hspace{0.03cm} |\hspace{0.05cm} x_{\mu})}
 \hspace{0.05cm}.$$

*The&nbsp; &raquo;'''sink entropy'''&laquo;&nbsp; $H(Y)$, the mean information content of the sink.&nbsp; $H(Y)$&nbsp; is the sum of the useful mutual information&nbsp; $I(X; Y)$&nbsp; and the useless irrelevance&nbsp; $H(Y|X)$, which comes exclusively from channel errors:
  	
:$$H(Y) = {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{1}{P_Y(Y)}\right ] \hspace{0.1cm}
= -{\rm E} \big [ {\rm log}_2 \hspace{0.1cm}{P_Y(Y)}\big ] \hspace{0.2cm} =I(X;Y) + H(Y|X) 
 \hspace{0.05cm}.$$

*The&nbsp; &raquo;'''joint entropy'''&laquo;&nbsp; $H(XY)$&nbsp; is the average information content of the 2D random quantity&nbsp; $XY$.&nbsp; It also describes an upper bound for the sum of source entropy and sink entropy:

:$$H(XY) = {\rm E} \left [ {\rm log} \hspace{0.1cm} \frac{1}{P_{XY}(X, Y)}\right ] = \sum_{\mu = 1}^{|X|}  \hspace{0.1cm} \sum_{\kappa = 1}^{|Y|} \hspace{0.1cm}
 P_{XY}(x_{\mu}\hspace{0.05cm}, y_{\kappa}) \cdot {\rm log} \hspace{0.1cm} \frac{1}{P_{XY}(x_{\mu}\hspace{0.05cm}, y_{\kappa})}\le H(X) + H(Y) \hspace{0.05cm}.$$ 

[[File:Transinf_2.png|right|frame|Considered model of the binary channel]]
{{GraueBox|TEXT=
$\text{Example 2}$:&nbsp; The same requirements as for&nbsp; [[Applets:Capacity_of_Memoryless_Digital_Channels#Zugrunde_liegendes_Modell_der_Digitalsignal.C3.BCbertragung|$\text{Example 1}$]] apply:&nbsp;

'''(1)'''&nbsp; The source symbols are not equally probable:
:$$P_X(X) = \big ( p_{\rm A},\ p_{\rm B} \big )=
\big ( 0.1,\ 0.9 \big )
\hspace{0.05cm}.$$ 
'''(2)'''&nbsp; Let the falsification probabilities be:  
:$$\begin{align*}p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}A}  & =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm a}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm A}) = 0.95\hspace{0.05cm},\hspace{0.8cm}p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}A}  =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm b}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm A}) = 0.05\hspace{0.05cm},\\
p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}B}  & =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm a}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm B}) = 0.40\hspace{0.05cm},\hspace{0.8cm}p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}B}  =  {\rm Pr}(Y\hspace{-0.1cm} = {\rm b}\hspace{0.05cm}\vert X \hspace{-0.1cm}= {\rm B}) = 0.60\end{align*}$$

:$$\Rightarrow \hspace{0.3cm}  P_{\hspace{0.01cm}Y\hspace{0.05cm} \vert \hspace{0.05cm}X}(Y\hspace{0.05cm}  \vert \hspace{0.05cm} X) = 
		\begin{pmatrix} 
		0.95  & 0.05\\
		0.4 & 0.6
		\end{pmatrix} \hspace{0.05cm}.$$

[[File:Inf_T_1_1_S4_vers2.png|frame|Binary entropy function as a function of&nbsp; $p$|right]]
*Because of condition&nbsp; '''(1)'''&nbsp; we obtain for the source entropy with the&nbsp; [[Information_Theory/Gedächtnislose_Nachrichtenquellen#Bin.C3.A4re_Entropiefunktion|$\text{binary entropy function}$]]&nbsp; $H_{\rm bin}(p)$:&nbsp;

:$$H(X) =  p_{\rm A} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm A}\hspace{0.1cm} } + p_{\rm B} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{p_{\rm B} }= H_{\rm bin} (p_{\rm A}) = H_{\rm bin} (0.1)= 0.469 \ {\rm bit}
\hspace{0.05cm};$$

::$$H_{\rm bin} (p) =  p \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p\hspace{0.1cm} } + (1 - p) \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{1 - p} \hspace{0.5cm}{\rm (unit\hspace{-0.15cm}: \hspace{0.15cm}bit\hspace{0.15cm}or\hspace{0.15cm}bit/symbol)}
\hspace{0.05cm}.$$

* Correspondingly, for the sink entropy with PMF&nbsp; $P_Y(Y) = \big ( p_{\rm a},\ p_{\rm b} \big )=
\big ( 0.455,\ 0.545 \big )$:
:$$H(Y) =   H_{\rm bin} (0.455)= 0.994 \ {\rm bit}
\hspace{0.05cm}.$$
*Next, we calculate the joint entropy:
:$$H(XY) =  p_{\rm Aa} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Aa}\hspace{0.1cm} }+ p_{\rm Ab} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Ab}\hspace{0.1cm} }+p_{\rm Ba} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Ba}\hspace{0.1cm} }+ p_{\rm Bb} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Bb}\hspace{0.1cm} }$$
:$$\Rightarrow \hspace{0.3cm}H(XY) =    0.095 \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{0.095 } +0.005 \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{0.005 }+0.36 \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{0.36 }+0.54 \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{0.54 }= 1.371 \ {\rm bit}
\hspace{0.05cm}.$$

According to the upper left diagram, the remaining information-theoretic quantities are thus also computable:
[[File:Transinf_4.png|right|frame|Information-theoretic model for&nbsp; $\text{Example 2}$]]

*the&nbsp; &raquo;'''equivocation'''&laquo;:
 
:$$H(X \vert Y) \hspace{-0.01cm} =\hspace{-0.01cm}  H(XY) \hspace{-0.01cm} -\hspace{-0.01cm}  H(Y) \hspace{-0.01cm}  = \hspace{-0.01cm}  1.371\hspace{-0.01cm}  -\hspace{-0.01cm}  0.994\hspace{-0.01cm} =\hspace{-0.01cm}  0.377\ {\rm bit}
 \hspace{0.05cm},$$

**the &raquo;'''irrelevance'''&laquo;:
 
:$$H(Y \vert X) = H(XY) - H(X)  = 1.371 - 0.994 = 0.902\ {\rm bit}
 \hspace{0.05cm}.$$

*the&nbsp;&raquo; '''mutual information'''&laquo;&nbsp;:
 
:$$I(X;Y) = H(X) + H(Y) - H(XY)  = 0.469 + 0.994 - 1.371 = 0.092\ {\rm bit}
 \hspace{0.05cm},$$

The results are summarized in the graph. 

''Note'':&nbsp; Equivocation and irrelevance could also be computed (but with extra effort) directly from the corresponding probability functions, for example:
  
:$$H(Y \vert X) = \hspace{-0.2cm} \sum_{(x, y) \hspace{0.05cm}\in \hspace{0.05cm}XY} \hspace{-0.2cm} P_{XY}(x,\hspace{0.05cm}y) \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{P_{\hspace{0.05cm}Y\hspace{-0.01cm}\vert \hspace{0.03cm}X}
(\hspace{0.05cm}y\hspace{0.03cm} \vert \hspace{0.05cm} x)}= p_{\rm Aa} \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}A} } +
p_{\rm Ab} \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}A} } +
p_{\rm Ba} \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}B} } +
p_{\rm Bb} \cdot {\rm log}_2 \hspace{0.1cm} \frac{1}{p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}B} } = 0.902 \ {\rm bit} \hspace{0.05cm}.$$}}


[[File:Transinf_3.png|right|frame|Considered model of the ternary channel:<br>Red transitions represent&nbsp; $p_{\rm a\hspace{0.03cm}\vert \hspace{0.03cm}A} = p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}B} = p_{\rm c\hspace{0.03cm}\vert \hspace{0.03cm}C} = q$&nbsp; and blue ones for&nbsp; $p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}A} = p_{\rm c\hspace{0.03cm}\vert \hspace{0.03cm}A}  =\text{...}= p_{\rm b\hspace{0.03cm}\vert \hspace{0.03cm}C}= (1-q)/2$]]
{{GraueBox|TEXT=
$\text{Example 3}$:&nbsp; Now we consider a transmission system with&nbsp; $M_X = M_Y = M=3$.&nbsp;

'''(1)'''&nbsp; Let the source symbols be equally probable:
:$$P_X(X) = \big ( p_{\rm A},\ p_{\rm B},\ p_{\rm C} \big )=
\big ( 1/3,\ 1/3,\ 1/3 \big )\hspace{0.30cm}\Rightarrow\hspace{0.30cm}H(X)={\rm log_2}\hspace{0.1cm}3 \approx 1.585 \ {\rm bit}
\hspace{0.05cm}.$$
'''(2)'''&nbsp; The channel model is symmetric &nbsp; &rArr; &nbsp; the sink symbols are also equally probable:
:$$P_Y(Y) = \big ( p_{\rm a},\ p_{\rm b},\ p_{\rm c} \big )=
\big ( 1/3,\ 1/3,\ 1/3 \big )\hspace{0.30cm}\Rightarrow\hspace{0.30cm}H(Y)={\rm log_2}\hspace{0.1cm}3 \approx 1.585 \ {\rm bit}
\hspace{0.05cm}.$$
'''(3)'''&nbsp; The joint probabilities are obtained as follows:
:$$p_{\rm Aa}= p_{\rm Bb}= p_{\rm Cc}= q/M,$$
:$$p_{\rm Ab}= p_{\rm Ac}= p_{\rm Ba}= p_{\rm Bc} = p_{\rm Ca}= p_{\rm Cb} = (1-q)/(2M)$$
:$$\Rightarrow\hspace{0.30cm}H(XY) =  3 \cdot p_{\rm Aa} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Aa}\hspace{0.1cm}  }+6 \cdot p_{\rm Ab} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p_{\rm Ab}\hspace{0.1cm} }= \
\text{...} \ = q \cdot {\rm log_2}\hspace{0.1cm}\frac{M}{q }+ (1-q) \cdot {\rm log_2}\hspace{0.1cm}\frac{M}{(1-q)/2 }.$$
[[File:Transinf_10.png|right|frame|Some results for&nbsp; $\text{Example 3}$]] 
'''(4)'''&nbsp; For the mutual information we get after some transformations considering the equation&nbsp;
:$$I(X;Y) = H(X) + H(Y) - H(XY)\text{:}$$
:$$\Rightarrow \hspace{0.3cm} I(X;Y) = {\rm log_2}\ (M) - (1-q) -H_{\rm bin}(q).$$
* For error-free ternary transfer&nbsp; $(q=1)$&nbsp; holds&nbsp; $I(X;Y) = H(X) = H(Y)={\rm log_2}\hspace{0.1cm}3$.
* With&nbsp; $q=0.8$&nbsp; the mutual information already decreases to&nbsp; $I(X;Y) = 0.663$&nbsp; and with&nbsp; $q=0.5$&nbsp; to&nbsp; $0.085$&nbsp; bit.
*The worst case from the point of view of information theory is&nbsp; $q=1/3$&nbsp; &rArr; &nbsp; $I(X;Y) = 0$.
*On the other hand, the worst case from the point of view of transmission theory is&nbsp; $q=0$&nbsp; &rArr; &nbsp; "not a single transmission symbol arrives correctly"&nbsp; is not so bad from the point of view of information theory.
* In order to be able to use this good result, however, channel coding is required at the transmitting end. }}
<br><br>
===Definition and meaning of channel capacity === 

If one calculates the mutual information&nbsp; $I(X, Y)$&nbsp; as explained in&nbsp; $\text{Example 2}$,&nbsp; then this depends not only on the discrete memoryless channel&nbsp; (DMC),&nbsp; but also on the source statistic   &nbsp;  ⇒  &nbsp;  $P_X(X)$&nbsp; Ergo: &nbsp; '''The mutual information'''&nbsp; $I(X, Y)$&nbsp;''' is not a pure channel characteristic'''.

{{BlaueBox|TEXT=
$\text{Definition:}$&nbsp; The&nbsp; &raquo;'''channel capacity'''&laquo;&nbsp; introduced by&nbsp; [https://en.wikipedia.org/wiki/Claude_Shannon $\text{Claude E. Shannon}$]&nbsp; according to his standard work&nbsp; [Sha48]<ref name = ''Sha48''>Shannon, C.E.: ''A Mathematical Theory of Communication''. In: Bell Syst. Techn. J. 27 (1948), S. 379-423 und S. 623-656.</ref>:
 
:$$C = \max_{P_X(X)} \hspace{0.15cm}  I(X;Y)  \hspace{0.05cm}.$$

The additional unit&nbsp; "bit/use"&nbsp; is often added.&nbsp; Since according to this definition the best possible source statistics are always the basis:
*$C$&nbsp; depends only on the channel properties &nbsp; ⇒ &nbsp; $P_{Y \vert X}(Y \vert X)$ &nbsp; but not on the source statistics &nbsp; ⇒ &nbsp; $P_X(X)$.&nbsp; }}


Shannon needed the quantity&nbsp; $C$&nbsp; to formulate the&nbsp; "Channel Coding Theorem" – one of the highlights of the information theory he founded.

{{BlaueBox|TEXT=
$\text{Shannon's Channel Coding Theorem:}$&nbsp; 
*For every transmission channel with channel capacity&nbsp; $C > 0$,&nbsp; there exists (at least) one&nbsp; $(k,\ n)$–block code,&nbsp; whose (block) error probability approaches zero&nbsp; as long as the code rate&nbsp; $R = k/n$&nbsp; is less than or equal to the channel capacity: &nbsp; 
:$$R ≤ C.$$
* The prerequisite for this, however,&nbsp;  is that the following applies to the block length of this code: &nbsp; $n → ∞.$


$\text{Reverse of Shannon's channel coding theorem:}$&nbsp;
 
If the rate&nbsp;  $R$&nbsp; of the&nbsp; $(n$,&nbsp; $k)$ block code used is larger than the channel capacity&nbsp; $C$,&nbsp; then an arbitrarily small block error probability can never be achieved.}}


[[File:Transinf_9.png|right|frame|Information-theoretic quantities for <br>different&nbsp; $p_{\rm A}$&nbsp; and&nbsp; $p_{\rm B}= 1- p_{\rm A}$ ]]
{{GraueBox|TEXT=
$\text{Example 4}$:&nbsp; We consider the same discrete memoryless channel as in &nbsp;$\text{Example 2}$.&nbsp;
In this&nbsp;$\text{Example 2}$&nbsp; the symbol probabilities&nbsp; $p_{\rm A} = 0.1$&nbsp; and&nbsp; $p_{\rm B}= 1- p_{\rm A}=0.9$&nbsp; were assumed.&nbsp; Thus the mutual information resulted in&nbsp;  $I(X;Y)= 0.092$&nbsp; bit/channel access &nbsp; &rArr; &nbsp; see first row, fourth column in the table.

The&nbsp; &raquo;'''channel capacity'''&laquo;&nbsp; is the mutual information&nbsp; $I(X, Y)$&nbsp; with best possible symbol probabilities&nbsp; $p_{\rm A} = 0.55$&nbsp; and&nbsp; $p_{\rm B}= 1- p_{\rm A}=0.45$:
:$$C = \max_{P_X(X)} \hspace{0.15cm}  I(X;Y) = 0.284 \ \rm bit/channel access \hspace{0.05cm}.$$

From the table you can see further&nbsp; (we do without the additional unit "bit/channel access" in the following):
*The parameter&nbsp; $p_{\rm A} = 0.1$&nbsp; was chosen very unfavorably, because with the present channel the symbol&nbsp; $\rm A$&nbsp; is more falsified than&nbsp; $\rm B$.&nbsp; Already with&nbsp; $p_{\rm A} = 0.9$&nbsp; results in a somewhat better value:&nbsp; $I(X; Y)=0.130$.
*For the same reason&nbsp; $p_{\rm A} = 0.55$,&nbsp; $p_{\rm B} = 0.45$&nbsp; gives a slightly better result than equally probable symbols&nbsp; $p_{\rm A} = p_{\rm B} =0.5$.
*The more asymmetric the channel is, the more the optimal probability function&nbsp; $P_X(X)$&nbsp; deviates from the uniform distribution.&nbsp; Conversely:&nbsp; If the channel is symmetric, the uniform distribution is always obtained.}} 


The ternary channel of &nbsp;$\text{Example 3}$&nbsp; is symmetric.&nbsp; Therefore here&nbsp; $P_X(X) = \big ( 1/3,\ 1/3,\ 1/3 \big )$&nbsp; is optimal for each&nbsp; $q$&ndash;value, and the mutual information&nbsp;  $I(X;Y)$&nbsp; given in the result table is at the same time the channel capacity&nbsp;  $C$.  




==Exercises==

*First, select the number&nbsp; $(1,\ 2,  \text{...} \ )$&nbsp; of the task to be processed.&nbsp; The number&nbsp; "$0$"&nbsp; corresponds to a "Reset":&nbsp; Same setting as at program start.
*A task description is displayed.&nbsp; The parameter values are adjusted.&nbsp; Solution after pressing "Show Solution".
*Source symbols are denoted by uppercase letters&nbsp; (binary:&nbsp; $\rm A$,&nbsp; $\rm B$),&nbsp; sink symbols by lowercase letters&nbsp; ($\rm a$,&nbsp; $\rm b$).&nbsp; Error-free transmission:&nbsp; $\rm A \rightarrow a$. 
*For all entropy values, the unit "bit/use" would have to be added.


{{BlueBox|TEXT=
'''(1)'''&nbsp; Let&nbsp; $p_{\rm A} = p_{\rm B} = 0.5$&nbsp; and&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B} = 0.1$.&nbsp; What is the channel model?&nbsp; What are the entropies&nbsp; $H(X), \, H(Y)$&nbsp; and the mutual information&nbsp; $I(X;\, Y)$?}}

:*&nbsp; Considered is the BSC model&nbsp; (Binary Symmetric Channel).&nbsp; Because of&nbsp; $p_{\rm A} = p_{\rm B} = 0.5$&nbsp; it holds for the entropies:&nbsp; $H(X) = H(Y) = 1$.
:*&nbsp; Because of&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B} = 0.1$&nbsp; eqivocation and irrelevance are also equal:&nbsp; $H(X \vert Y) = H(Y \vert X) = H_{\rm bin}(p_{\rm b \vert A}) = H_{\rm bin}(0.1) =0.469$.
:*&nbsp; The mutual information is&nbsp; $I(X;\, Y) = H(X) - H(X \vert Y)= 1-H_{\rm bin}(p_{\rm b \vert A}) = 0.531$&nbsp; and the joint entropy is&nbsp; $H(XY) =1.469$. 
  
{{BlueBox|TEXT=
'''(2)'''&nbsp; Let further&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B} = 0.1$, but now the symbol probability is&nbsp; $p_{\rm A} = 0. 9$.&nbsp; What is the capacity&nbsp; $C_{\rm BSC}$&nbsp; of the BSC channel with&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B}$?<br>&nbsp; &nbsp; &nbsp; &nbsp; 
Which&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B}$&nbsp; leads to the largest possible channel capacity and which&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B}$&nbsp; leads to the channel capacity&nbsp; $C_{\rm BSC}=0$?}}

:*&nbsp; The capacity&nbsp; $C_{\rm BSC}$&nbsp; is equal to the maximum mutual information&nbsp; $I(X;\, Y)$&nbsp; considering the optimal symbol probabilities.
:*&nbsp; Due to the symmetry of the BSC model&nbsp; equally probable symbols&nbsp; $(p_{\rm A} = p_{\rm B} = 0.5)$&nbsp; lead to the optimum &nbsp; &rArr; &nbsp; $C_{\rm BSC}=0.531$.
:*&nbsp; The best is the&nbsp; "ideal channel"&nbsp; $(p_{\rm b \vert A} = p_{\rm a \vert B} = 0)$&nbsp; &rArr; &nbsp; $C_{\rm BSC}=1$. &nbsp; The worst BSC channel results with&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B} = 0.5$ &nbsp; &rArr; &nbsp; $C_{\rm BSC}=0$.
:*&nbsp; But also with &nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B} = 1$&nbsp; we get&nbsp; $C_{\rm BSC}=1$.&nbsp; Here all symbols are inverted, which is information theoretically the same as&nbsp; $\langle Y_n \rangle \equiv \langle X_n \rangle$. 

{{BlueBox|TEXT=
'''(3)'''&nbsp; Let&nbsp; $p_{\rm A} = p_{\rm B} = 0.5$,&nbsp; $p_{\rm b \vert A} = 0.05$&nbsp; and&nbsp; $ p_{\rm a \vert B} = 0.4$. &nbsp; Interpret the results in comparison to the experiment&nbsp; $(1)$&nbsp; and to the&nbsp; $\text{example 2}$&nbsp; in the theory section.}}

:*&nbsp; Unlike the experiment&nbsp; $(1)$&nbsp; no BSC channel is present here.&nbsp; Rather, the channel considered here is asymmetric:&nbsp; $p_{\rm b \vert A} \ne p_{\rm a \vert B}$.
:*&nbsp; According to&nbsp; $\text{Example 2}$&nbsp; it holds for&nbsp; $p_{\rm A} = 0.1,\ p_{\rm B} = 0.9$: &nbsp; &nbsp; $H(X)= 0.469$,&nbsp; $H(Y)= 0.994$,&nbsp; $H(X \vert Y)=0.377$,&nbsp; $H(Y \vert X)=0.902$,&nbsp; $I(X;\vert Y)=0.092$.
:*&nbsp; Now it holds&nbsp; $p_{\rm A} = p_{\rm B} = 0.5$&nbsp; and we get&nbsp; $H(X)=1,000$,&nbsp; $H(Y)=0.910$,&nbsp; $H(X \vert Y)=0.719$,&nbsp; $H(Y \vert X)=0.629$,&nbsp; $I(X;\ Y)=0.281$.
:*&nbsp; All output values depend significantly on&nbsp; $p_{\rm A}$&nbsp; and&nbsp; $p_{\rm B}=1-p_{\rm A}$&nbsp; except for the conditional probabilities&nbsp; ${\rm Pr}(Y \vert X)\in \{\hspace{0.05cm}0.95,\ 0.05,\ 0.4,\ 0.6\hspace{0.05cm} \}$.

{{BlueBox|TEXT=
'''(4)'''&nbsp; Let further&nbsp; $p_{\rm A} = p_{\rm B}$,&nbsp; $p_{\rm b \vert A} = 0.05$,&nbsp; $ p_{\rm a \vert B} = 0.4$.&nbsp; What differences do you see in terms of analytical calculation and "simulation"&nbsp; $(N=10000)$.}}
   
:*&nbsp; The joint probabilities are&nbsp; $p_{\rm Aa} =0.475$,&nbsp; $p_{\rm Ab} =0.025$,&nbsp; $p_{\rm Ba} =0.200$,&nbsp;$p_{\rm Bb} =0.300$.&nbsp; Simulation:&nbsp; Approximation by relative frequencies:
:*&nbsp; For example, for&nbsp; $N=10000$:&nbsp; $h_{\rm Aa} =0.4778$,&nbsp; $h_{\rm Ab} =0.0264$,&nbsp; $h_{\rm Ba} =0.2039$,&nbsp;$h_{\rm Bb} =0.2919$.&nbsp; After pressing&nbsp; "New sequence"&nbsp; slightly different values.
:*&nbsp; For all subsequent calculations, no principal difference between theory and simulation, except&nbsp; $p \to h$.&nbsp; Examples:&nbsp;
:*&nbsp; $p_{\rm A} = 0.5 \to h_{\rm A}=h_{\rm Aa} + h_{\rm Ab} =0.5042$,&nbsp; $p_b = 0.325 \to h_{\rm b}=h_{\rm Ab} + h_{\rm Bb} =0. 318$,&nbsp; $p_{b|A} = 0.05 \to h_{\rm b|A}=h_{\rm Ab}/h_{\rm A} =0.0264/0.5042= 0.0524$,&nbsp;
:*&nbsp; $p_{\rm A|b} = 0.0769 \to h_{\rm A|b}=h_{\rm Ab}/h_{\rm b} =0.0264/0.318= 0.0830$.&nbsp; Thus, this simulation yields&nbsp; $I_{\rm Sim}(X;\ Y)=0.269$&nbsp; instead of&nbsp; $I(X;\ Y)=0.281$.

{{BlueBox|TEXT=
'''(5)'''&nbsp; Setting according to&nbsp; $(4)$.&nbsp; How does&nbsp; $I_{\rm Sim}(X;\ Y)$&nbsp; differ  from&nbsp; $I(X;\ Y) = 0.281$&nbsp; for&nbsp; $N=10^3$,&nbsp; $10^4$,&nbsp; $10^5$&nbsp;?&nbsp; In each case, averaging over ten realizations.  }}  
:*&nbsp; $N=10^3$: &nbsp; $0.232 \le I_{\rm Sim} \le 0.295$, &nbsp; mean:&nbsp; $0.263$ &nbsp; # &nbsp; $N=10^4$: &nbsp; $0.267 \le I_{\rm Sim} \le 0.293$, &nbsp; mean:&nbsp; $0.279$ &nbsp; # &nbsp; $N=10^5$: &nbsp; $0.280 \le I_{\rm Sim} \le 0.285$ &nbsp; mean:&nbsp; $0.282$.
:*&nbsp; With&nbsp; $N=10^6$&nbsp; for this channel, the simulation result differs from the theoretical value by less than&nbsp; $\pm 0.001$.&nbsp;

{{BlueBox|TEXT=
'''(6)'''&nbsp; What is the capacity&nbsp; $C_6$&nbsp; of this channel with&nbsp; $p_{\rm b \vert A} = 0.05$,&nbsp; $ p_{\rm a \vert B} = 0.4$?&nbsp;  Is the error probability&nbsp; $0$&nbsp; possible with the code rate&nbsp; $R=0.3$? }}

:*&nbsp; $C_6=0.284$&nbsp; is the maximum of&nbsp; $I(X;\ Y)$&nbsp; for &nbsp; $p_{\rm A} =0.55$&nbsp; &rArr; &nbsp;$p_{\rm B} =0. 45$.&nbsp; Simulation over&nbsp; ten times&nbsp; $N=10^5$:&nbsp; $0.281 \le I_{\rm Sim}(X;\ Y) \le 0.289$.
:*&nbsp; With the code rate&nbsp; $R=0.3 > C_6$&nbsp; an arbitrarily small block error probability is not achievable even with the best possible coding.

{{BlueBox|TEXT=
'''(7)'''&nbsp; Now let&nbsp; $p_{\rm A} = p_{\rm B}$,&nbsp; $p_{\rm b \vert A} = 0$,&nbsp; $ p_{\rm a \vert B} = 0.5$. &nbsp; What property does this asymmetric channel exhibit?&nbsp; What values result for&nbsp; $H(X)$,&nbsp; $H(X \vert Y)$,&nbsp; $I(X;\ Y)$&nbsp;?  }}

:*&nbsp; The symbol&nbsp; $\rm A$&nbsp; is never falsified, the symbol&nbsp; $\rm B$&nbsp; with (information theoretically) maximum falsification probability&nbsp; $ p_{\rm a \vert B} = 0.5$
:*&nbsp; The total falsification probability is&nbsp; $ {\rm Pr} (Y_n \ne X_n)= p_{\rm A} \cdot p_{\rm b \vert A} + p_{\rm B} \cdot p_{\rm a \vert B}= 0.25$&nbsp; &rArr; &nbsp; about&nbsp; $25\%$&nbsp; of the output sink symbols are "purple".
:*&nbsp; Joint probabilities:&nbsp; $p_{\rm Aa}= 1/2,\ p_{\rm Ab}= 0,\ p_{\rm Ba}= p_{\rm Bb}= 1/4$,&nbsp; &nbsp; Inference probabilities: &nbsp; $p_{\rm A \vert a}= 1,\ p_{\rm B \vert a}= 0,\ p_{\rm A \vert b}= 1/3,\ p_{\rm B \vert b}= 2/3$. 
:*&nbsp; From this we get for equivocation&nbsp; $H(X \vert Y)=0.689$; &nbsp; with source entropy&nbsp; $H(X)= 1$ &nbsp; &rArr; &nbsp; $I(X;\vert Y)=H(X)-H(X \vert Y)=0.311$.

{{BlueBox|TEXT=
'''(8)'''&nbsp; What is the capacity&nbsp; $C_8$&nbsp; of this channel with&nbsp; $p_{\rm b \vert A} = 0.05$,&nbsp; $ p_{\rm a \vert B} = 035$?&nbsp; Is the error probability&nbsp; $0$&nbsp; possible with the code rate&nbsp; $R=0.3$? }}

:*&nbsp; $C_8=0.326$&nbsp; is the maximum of&nbsp; $I(X;\ Y)$&nbsp; for &nbsp; $p_{\rm A} =0.55$.&nbsp; Thus, because of&nbsp; $C_8 >R=0.3 $&nbsp; an arbitrarily small block error probability is achievable.
:*&nbsp; The only difference compared to&nbsp; $(6)$ &nbsp; &rArr; &nbsp; $C_6=0.284 < 0.3$&nbsp; is the slightly smaller falsification probability&nbsp; $ p_{\rm a \vert B} = 0.35$&nbsp; instead of&nbsp; $ p_{\rm a \vert B} = 0.4$.

{{BlueBox|TEXT=
'''(9)'''&nbsp; We consider the ideal ternary channel:&nbsp; $p_{\rm a \vert A} = p_{\rm b \vert B}=p_{\rm c \vert C}=1$.&nbsp; What is its capacity&nbsp; $C_9$?&nbsp; What is the maximum mutual information displayed by the program? }}

:*&nbsp; Due to the symmetry of the channel model, equally probable symbols&nbsp; $(p_{\rm A} = p_{\rm B}=p_{\rm C}=1/3)$&nbsp; lead to the channel capacity:&nbsp; $C_9 = \log_2\ (3) = 1.585$.
:*&nbsp; Since in the program all parameter values can only be entered with a resolution of&nbsp; $0.05$&nbsp;, for&nbsp; $I(X;\ Y)$&nbsp; this maximum value is not reached.
:*&nbsp; Possible approximations:&nbsp; $p_{\rm A} = p_{\rm B}= 0.3, \ p_{\rm C}=0.4$&nbsp; &rArr; &nbsp; $I(X;\ Y)= 1. 571$ &nbsp; &nbsp; # &nbsp; &nbsp; $p_{\rm A} = p_{\rm B}= 0.35, \ p_{\rm C}=0.3$&nbsp; &rArr; &nbsp; $I(X;\ Y)= 1.581$.  
 
{{BlueBox|TEXT=
'''(10)'''&nbsp; Let the source symbols be (nearly) equally probable.&nbsp; Interpret the other settings and the results.  }}

:*&nbsp; The falsification probabilities are&nbsp; $p_{\rm b \vert A} = p_{\rm c \vert B}=p_{\rm a \vert C}=1$ &nbsp; &rArr; &nbsp; no single sink symbol is equal to the source symbol.
:*&nbsp; This cyclic mapping has no effect on the channel capacity:&nbsp; $C_{10} = C_9 = 1.585$.&nbsp; The program returns&nbsp; ${\rm Max}\big[I(X;\ Y)\big]= 1.581$.

{{BlueBox|TEXT=
'''(11)'''&nbsp; We consider up to and including&nbsp; $(13)$&nbsp; the same ternary source. &nbsp; What results are obtained for&nbsp; $p_{\rm b \vert A} = p_{\rm c \vert B}=p_{\rm a \vert C}=0.2$&nbsp; and&nbsp; $p_{\rm c \vert A} = p_{\rm a \vert B}=p_{\rm b \vert C}=0$?  }}
:*&nbsp; Each symbol can only be falsified into one of the two possible other symbols.&nbsp; From&nbsp; $p_{\rm b \vert A} = p_{\rm c \vert B}=p_{\rm a \vert C}=0.2$&nbsp; it follows&nbsp; $p_{\rm a \vert A} = p_{\rm b \vert B}=p_{\rm c \vert C}=0.8$.
:*&nbsp; This gives us for the maximum mutual information&nbsp; ${\rm Max}\big[I(X;\ Y)\big]= 0.861$&nbsp; and for the channel capacity a slightly larger value:&nbsp; $C_{11} \gnapprox 0.861$.

{{BlueBox|TEXT=
'''(12)'''&nbsp; How do the results change if each symbol is &nbsp; $80\%$&nbsp; transferred correctly and &nbsp; $10\%$&nbsp; falsified each in one of the other two symbols?    }}
:*&nbsp; Although the probability of correct transmission is with&nbsp; $80\%$&nbsp; as large as in&nbsp; '''(11)''', here the channel capacity&nbsp; $C_{12} \gnapprox 0.661$ is smaller.
:*&nbsp; If one knows for the channel&nbsp; $(11)$&nbsp; that&nbsp; $X = \rm A$&nbsp; has been falsified, one also knows&nbsp; $Y = \rm b$.&nbsp; But not for channel&nbsp; $(12)$&nbsp;  &rArr; &nbsp; the channel is less favorable.

{{BlueBox|TEXT=
'''(13)'''&nbsp; Let the falsification probabilities now be&nbsp; $p_{\rm b \vert A} = p_{\rm c \vert A} = p_{\rm a \vert B} = p_{\rm c \vert B}=p_{\rm a \vert C}=p_{\rm b \vert C}=0.5$.&nbsp; Interpret this redundancy-free ternary channel.   }}
:*&nbsp; No single sink symbol is equal to its associated source symbol; with respect to the other two symbols, a &nbsp;$50\hspace{-0.1cm}:\hspace{-0.1cm}50$&nbsp; decision must be made.
:*&nbsp; Nevertheless, here the channel capacity is &nbsp;$C_{13} \gnapprox 0.584$&nbsp; only slightly smaller than in the previous experiment:&nbsp; $C_{12} \gnapprox 0.661$.
:*&nbsp; The channel capacity&nbsp; $C=0$&nbsp; results for the redundancy-free ternary channel exactly for the case where all nine falsification probabilities are equal to&nbsp; $1/3$&nbsp;.

{{BlueBox|TEXT=
'''(14)'''&nbsp; What is the capacity $C_{14}$ of the ternary channel with&nbsp; $p_{\rm b \vert A} = p_{\rm a \vert B}= 0$&nbsp; and&nbsp; $p_{\rm c \vert A} = p_{\rm c \vert B} = p_{\rm a \vert C}=p_{\rm b \vert C}=0. 1$&nbsp; &rArr; &nbsp; $p_{\rm a \vert A} = p_{\rm b \vert B}=0.9$, &nbsp; $p_{\rm c \vert C} =0.8$?    }}
:*&nbsp; With the default&nbsp; $p_{\rm A}=p_{\rm B}=0.2$ &nbsp; &rArr; &nbsp; $p_{\rm C}=0.6$&nbsp; we get&nbsp; $I(X;\ Y)= 0.738$.&nbsp; Now we are looking for "better" symbol probabilities.
:*&nbsp; From the symmetry of the channel, it is obvious that&nbsp; $p_{\rm A}=p_{\rm B}$&nbsp; is optimal.&nbsp; The channel capacity&nbsp; $C_{14}=0.995$&nbsp; is obtained for&nbsp; $p_{\rm A}=p_{\rm B}=0.4$ &nbsp; &rArr; &nbsp; $p_{\rm C}=0.2$.
:*&nbsp; Example:&nbsp; Ternary transfer if the middle symbol&nbsp; $\rm C$&nbsp; can be distorted in two directions, but the outer symbols can only be distorted in one direction at a time.
<br><br>

==Applet Manual==
[[File:Anleitung_transinformation.png|left|600px]]
<br><br><br><br>
&nbsp; &nbsp; '''(A)''' &nbsp; &nbsp; Auswahlmöglichkeit, ob &nbsp;"analytisch"&nbsp; oder &nbsp;"per Simulation"

&nbsp; &nbsp; '''(B)''' &nbsp; &nbsp; Einstellung des Parameters&nbsp; $N$&nbsp; für die Simulation

&nbsp; &nbsp; '''(C)''' &nbsp; &nbsp; Auswahlmöglichkeit, ob &nbsp;"Binärquelle"&nbsp; oder &nbsp;"Ternärquelle"

&nbsp; &nbsp; '''(D)''' &nbsp; &nbsp; Einstellung der Symbolwahrscheinlichkeiten

&nbsp; &nbsp; '''(E)''' &nbsp; &nbsp; Einstellung der Übergangswahrscheinlichkeiten

&nbsp; &nbsp; '''(F)''' &nbsp; &nbsp; Numerikausgabe verschiedener Wahrscheinlichkeiten 

&nbsp; &nbsp; '''(G)''' &nbsp; &nbsp; Zwei Schaubilder mit den informationstheoretischen Größen

&nbsp; &nbsp; '''(H)''' &nbsp; &nbsp; Ausgabe einer beispielhaften Quellensymbolfolge

&nbsp; &nbsp; '''(I)''' &nbsp; &nbsp; Zugehörige simulierte Sinkensymbolfolge

&nbsp; &nbsp; '''(J)''' &nbsp; &nbsp; Bereich für Übungen:  Aufgabenauswahl, Fragen, Musterlösungen
<br clear=all>

==About the Authors==
<br>
This interactive calculation tool was designed and implemented at the&nbsp; [https://www.ei.tum.de/en/lnt/home/ $\text{Institute for Communications Engineering}$]&nbsp; at the&nbsp; [https://www.tum.de/en Technical University of Munich]. 
*The first version was created in 2010 by&nbsp; [[Biografien_und_Bibliografien/An_LNTwww_beteiligte_Studierende#Martin_V.C3.B6lkl_.28Diplomarbeit_LB_2010.29|$\text{Martin Völkl}$]]&nbsp; as part of his diploma thesis with “FlashMX – Actionscript”&nbsp; (Supervisor:&nbsp; [[Biografien_und_Bibliografien/An_LNTwww_beteiligte_Mitarbeiter_und_Dozenten#Prof._Dr.-Ing._habil._G.C3.BCnter_S.C3.B6der_.28am_LNT_seit_1974.29|$\text{Günter Söder}$]]&nbsp; and&nbsp; [[Biografien_und_Bibliografien/An_LNTwww_beteiligte_Mitarbeiter_und_Dozenten#Dr.-Ing._Klaus_Eichin_.28am_LNT_von_1972-2011.29|$\text{Klaus Eichin}$]]).
 
*In 2020 the program was redesigned via HTML5/JavaScript by&nbsp; [[Biographies_and_Bibliographies/An_LNTwww_beteiligte_Studierende#Veronika_Hofmann_.28Ingenieurspraxis_Math_2020.29|$\text{Veronika Hofmann}$]]&nbsp;  (Ingenieurspraxis Mathematik, Supervisor:&nbsp; [[Biographies_and_Bibliographies/LNTwww_members_from_LÜT#Benedikt_Leible.2C_M.Sc._.28at_L.C3.9CT_since_2017.29|$\text{Benedikt Leible}$]]&nbsp; and&nbsp; [[Biographies_and_Bibliographies/Beteiligte_der_Professur_Leitungsgebundene_%C3%9Cbertragungstechnik#Tasn.C3.A1d_Kernetzky.2C_M.Sc._.28bei_L.C3.9CT_seit_2014.29|Tasnád Kernetzky]] ).

*Last revision and English version 2021 by&nbsp; [[Biografien_und_Bibliografien/An_LNTwww_beteiligte_Studierende#Carolin_Mirschina_.28Ingenieurspraxis_Math_2019.2C_danach_Werkstudentin.29|$\text{Carolin Mirschina}$]]&nbsp; in the context of a working student activity.&nbsp; Translation using DEEPL.com (free version).

*The conversion of this applet was financially supported by&nbsp; [https://www.ei.tum.de/studium/studienzuschuesse/ $\text{Studienzuschüsse}$]&nbsp; (TUM Department of Electrical and Computer Engineering).&nbsp; We thank.



==Once again: Open Applet in new Tab==

{{LntAppletLinkEnDe|transinformation_en|transinformation}}