Exercise 4.1: Log Likelihood Ratio

Considered channel models

To interpret the "log likelihood ratio" $\rm (LLR)$ we start from the "binary symmetric channel" $\rm (BSC)$ as in the "theory section" .

For the binary random variables at the channel input and output holds:

$$x \in \{0\hspace{0.05cm}, 1\} \hspace{0.05cm},\hspace{0.25cm}y \in \{0\hspace{0.05cm}, 1\} \hspace{0.05cm}. $$

This model is shown in the upper graph. The following applies to the conditional probabilities in the forward direction:

$${\rm Pr}(y = 1\hspace{0.05cm}|\hspace{0.05cm} x = 0) = {\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 1) = \varepsilon \hspace{0.05cm},$$

$${\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 0) = {\rm Pr}(y = 1\hspace{0.05cm}|\hspace{0.05cm} x = 1) = 1-\varepsilon \hspace{0.05cm}.$$

The falsification probability $\varepsilon$ is the crucial parameter of the BSC model.

Regarding the probability distribution at the input instead of considering the probabilities ${\rm Pr}(x = 0)$ and ${\rm Pr}(x = 1)$ it is convenient to consider the log likelihood ratio.

For the unipolar approach used here, the following applies by definition:

$$L_{\rm A}(x)={\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0)}{{\rm Pr}(x = 1)}\hspace{0.05cm},$$

where the subscript $\rm A$ indicates the "a-priori log likelihood ratio" or the "a-priori L–value".

For example, for ${\rm Pr}(x = 0) = 0.2 \ \Rightarrow \ {\rm Pr}(x = 1) = 0.8$ ⇒ $L_{\rm A}(x) = \, -1.382$.

From the BSC model, it is possible to determine the L–value of the conditional probabilities ${\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x)$ in forward direction $($German: "Vorwärtsrichtung" ⇒ subscript "V"$)$, which is denoted by $L_{\rm V}(y)$ in the present exercise:

$$L_{\rm V}(y) = L(y\hspace{0.05cm}|\hspace{0.05cm}x) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x = 0)}{{\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x = 1)} = \left\{ \begin{array}{c} {\rm ln} \hspace{0.15cm} [(1 - \varepsilon)/\varepsilon]\\ {\rm ln} \hspace{0.15cm} [\varepsilon/(1 - \varepsilon)] \end{array} \right.\hspace{0.15cm} \begin{array}{*{1}c} {\rm f\ddot{u}r} \hspace{0.15cm} y = 0, \\ {\rm f\ddot{u}r} \hspace{0.15cm} y = 1. \\ \end{array}$$

For example, for $\varepsilon = 0.1$:

$$L_{\rm V}(y = 0) = +2.197\hspace{0.05cm}, \hspace{0.3cm}L_{\rm V}(y = 1) = -2.197\hspace{0.05cm}.$$

Of particular importance to coding theory are the inference probabilities ${\rm Pr}(x\hspace{0.05cm}|\hspace{0.05cm}y)$, which are related to the forward probabilities ${\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x)$ and the input probabilities ${\rm Pr}(x = 0)$ and ${\rm Pr}(x = 1)$ via Bayes' theorem.

The corresponding L–value in forward direction $($German: "Vorwärtsrichtung" ⇒ subscript "V"$)$ in this exercise is denoted by $L_{\rm R}(y)$ :

$$L_{\rm R}(y) = L(x\hspace{0.05cm}|\hspace{0.05cm}y) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0)\hspace{0.05cm}|\hspace{0.05cm}y)}{{\rm Pr}(x = 1)\hspace{0.05cm}|\hspace{0.05cm}y)} \hspace{0.05cm} .$$

Hints:

The exercise belongs to the chapter "Soft–in Soft–out Decoder".
Reference is made in particular to the page "Reliability Information – Log Likelihood Ratio".
In the last subtasks we have to clarify whether the found relations between $L_{\rm A}, \ L_{\rm V}$ and $L_{\rm R}$ can also be transferred to the "2 on $M$ channel".
For this purpose, we choose a bipolar approach for the input symbols: "$0$" → "$+1$" and "$1$" → "$–1$".

Questions

Solution

(1) For the conditional probabilities, according to the "Bayes' theorem" with intersection $A ∩ B$:

$${\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm} A) = \frac{{\rm Pr}(A \cap B)}{{\rm Pr}(A)}\hspace{0.05cm}, \hspace{0.3cm} {\rm Pr}(A \hspace{0.05cm}|\hspace{0.05cm} B) = \frac{{\rm Pr}(A \cap B)}{{\rm Pr}(B)}\hspace{0.3cm} \Rightarrow \hspace{0.3cm}{\rm Pr}(A \hspace{0.05cm}|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm} A) \cdot \frac{{\rm Pr}(A)}{{\rm Pr}(B)}\hspace{0.05cm}.$$

Correct is the proposition 3. In the special case ${\rm Pr}(B) = {\rm Pr}(A)$ also the suggestion 1 would be correct.

(2) With $A$ ⇒ "$x = 0$" and $B$ ⇒ "$y = 0$" we immediately get the equation according to proposition 1:

$${\rm Pr}(x = 0\hspace{0.05cm}|\hspace{0.05cm} y = 0) = {\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 0) \cdot \frac{{\rm Pr}(x = 0)}{{\rm Pr}(y = 0)}\hspace{0.05cm}.$$

(3) We compute the $L$ value of the inference probabilities. Assuming $y = 0$ holds:

$$L_{\rm R}(y= 0) \hspace{-0.15cm} \ = \ \hspace{-0.15cm} L(x\hspace{0.05cm}|\hspace{0.05cm}y= 0)= {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0\hspace{0.05cm}|\hspace{0.05cm}y=0)}{{\rm Pr}(x = 1\hspace{0.05cm}|\hspace{0.05cm}y=0)} = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x=0) \cdot {\rm Pr}(x = 0) / {\rm Pr}(y = 0)}{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x = 1)\cdot {\rm Pr}(x = 1) / {\rm Pr}(y = 0)} $$

$$\Rightarrow \hspace{0.3cm} L_{\rm R}(y= 0)= {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x=0) }{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x = 1)} + {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x=0) }{{\rm Pr}(x = 1)}$$

$$\Rightarrow \hspace{0.3cm} L_{\rm R}(y= 0) = L(x\hspace{0.05cm}|\hspace{0.05cm}y= 0) = L_{\rm V}(y= 0) + L_{\rm A}(x)\hspace{0.05cm}.$$

Similarly, assuming $y = 1$, the result is:

$$L_{\rm R}(y= 1) = L(x\hspace{0.05cm}|\hspace{0.05cm}y= 1) = L_{\rm V}(y= 1) + L_{\rm A}(x)\hspace{0.05cm}.$$

The two results can be summarized using $y ∈ \{0, \, 1\}$ and.

the input LLR,

$$L_{\rm A}(x) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x=0) }{\rm Pr}(x = 1)}\hspace{0.05cm},$$

as well as the forward LLR,

$$L_{\rm V}(y) = L(y\hspace{0.05cm}|\hspace{0.05cm}x) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y \hspace{0.05cm}|\hspace{0.05cm}x=0) }{{\rm Pr}(y \hspace{0.05cm}|\hspace{0.05cm}x = 1)} \hspace{0.05cm},$$

as follows:

$$L_{\rm R}(y) = L(x\hspace{0.05cm}|\hspace{0.05cm}y) = L_{\rm V}(y) + L_{\rm A}(x)\hspace{0.05cm}.$$

The identity $L_{\rm R}(y) ≡ L_{\rm V}(y)$ requires $L_{\rm A}(x) = 0$ ⇒ equally probable symbols ⇒ proposition 2.

(4) From the exercise description, you can see that with corruption probability $\varepsilon = 0.1$, the initial value $y = 1$ leads to forward–LLR $L_{\rm V}(y = 1) = \, –2.197$.

Wegen ${\rm Pr}(x = 0) = 1/2 \ \Rightarrow \ L_{\rm A}(x) = 0$ gilt somit auch:

$$L_{\rm R}(y = 1) = L_{\rm V}(y = 1) \hspace{0.15cm}\underline{= -2.197}\hspace{0.05cm}.$$

(5) With the same corruption probability $\varepsilon = 0.1$ $L_{\rm V}(y = 0)$ differs from $L_{\rm V}(y = 1)$ only by the sign.

With ${\rm Pr}(x = 0) = 0.2 \ \Rightarrow \ L_{\rm A}(x) = \, -1.382$ we thus obtain:

$$L_{\rm R}(y = 0) = (+)2.197 - 1.382 \hspace{0.15cm}\underline{=+0.815}\hspace{0.05cm}.$$

(6) As I'm sure you'll be happy to verify, the relation $L_{\rm R} = L_{\rm V} + L_{\rm A}$ also holds for the "2 on $M$ channel", regardless of the size (Für Günter: Umfang?) $M$ of the output alphabet ⇒ Answer Yes.

(7) The AWGN channel is described by the outlined "2–on–$M$–channel" with $M → ∞$ also ⇒ Answer Yes.

	${\rm Pr}(A\hspace{0.05cm}\|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}\|\hspace{0.05cm} A)$,
	${\rm Pr}(A\hspace{0.05cm}\|\hspace{0.05cm}B) = {\rm Pr}(B\hspace{0.05cm}\|\hspace{0.05cm} A) \cdot {\rm Pr}(B) / {\rm Pr}(A)$,
	${\rm Pr}(A\hspace{0.05cm}\|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}\|\hspace{0.05cm}A) \cdot {\rm Pr}(A) / {\rm Pr}(B)$.

	${\rm Pr}(x = 0 \| y = 0) = {\rm Pr}(y = 0 \| x = 0) \cdot {\rm Pr}(x = 0) / {\rm Pr}(y = 0)$,
	${\rm Pr}(x = 0 \| y = 0) = {\rm Pr}(y = 0 \| x = 0) \cdot {\rm Pr}(y = 0) / {\rm Pr}(x = 0)$.

	For any input distribution ${\rm Pr}(x = 0), \ {\rm Pr}(x = 1)$.
	For the uniform distribution only: $\hspace{0.2cm} {\rm Pr}(x = 0) = {\rm Pr}(x = 1) = 1/2$.

	Yes.
	No.

	Yes.
	No.