Aufgaben:Exercise 4.1: Log Likelihood Ratio

• Return to book

Considered channel models

To interpret the "log likelihood ratio"  $\rm (LLR)$  we start from the  "binary symmetric channel"  $\rm (BSC)$  as in the  "theory section" .

For the binary random variables at the channel input and output holds:

$$x \in \{0\hspace{0.05cm}, 1\} \hspace{0.05cm},\hspace{0.25cm}y \in \{0\hspace{0.05cm}, 1\} \hspace{0.05cm}. $$

This model is shown in the upper graph.  The following applies to the conditional probabilities in the forward direction:

$${\rm Pr}(y = 1\hspace{0.05cm}|\hspace{0.05cm} x = 0) = {\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 1) = \varepsilon \hspace{0.05cm},$$

$${\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 0) = {\rm Pr}(y = 1\hspace{0.05cm}|\hspace{0.05cm} x = 1) = 1-\varepsilon \hspace{0.05cm}.$$

The falsification probability  $\varepsilon$  is the crucial parameter of the BSC model.

Regarding the probability distribution at the input instead of considering the probabilities  ${\rm Pr}(x = 0)$  and  ${\rm Pr}(x = 1)$  it is convenient to consider the  log likelihood ratio.

For the unipolar approach used here,  the following applies by definition:

$$L_{\rm A}(x)={\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0)}{{\rm Pr}(x = 1)}\hspace{0.05cm},$$

where the subscript  "$\rm A$"  indicates the  "a-priori log likelihood ratio"  or the  "a-priori L–value".

For example,  for  ${\rm Pr}(x = 0) = 0.2 \ \Rightarrow \ {\rm Pr}(x = 1) = 0.8$   ⇒   $L_{\rm A}(x) = \, -1.382$.

From the BSC model,  it is possible to determine the  L–value of the conditional probabilities  ${\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x)$  in forward direction  $($German:  "Vorwärtsrichtung"   ⇒   subscript "V"$)$,  which is denoted by  $L_{\rm V}(y)$  in the present exercise:

$$L_{\rm V}(y) = L(y\hspace{0.05cm}|\hspace{0.05cm}x) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x = 0)}{{\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x = 1)} = \left\{ \begin{array}{c} {\rm ln} \hspace{0.15cm} [(1 - \varepsilon)/\varepsilon]\\ {\rm ln} \hspace{0.15cm} [\varepsilon/(1 - \varepsilon)] \end{array} \right.\hspace{0.15cm} \begin{array}{*{1}c} {\rm f\ddot{u}r} \hspace{0.15cm} y = 0, \\ {\rm f\ddot{u}r} \hspace{0.15cm} y = 1. \\ \end{array}$$

For example,  for  $\varepsilon = 0.1$:

$$L_{\rm V}(y = 0) = +2.197\hspace{0.05cm}, \hspace{0.3cm}L_{\rm V}(y = 1) = -2.197\hspace{0.05cm}.$$

Of particular importance to coding theory are the inference probabilities  ${\rm Pr}(x\hspace{0.05cm}|\hspace{0.05cm}y)$,  which are related to the backward probabilities  ${\rm Pr}(y\hspace{0.05cm}|\hspace{0.05cm}x)$  and the input probabilities  ${\rm Pr}(x = 0)$  and  ${\rm Pr}(x = 1)$  via Bayes' theorem.

The corresponding    L–value in backward direction  $($German:  "Rückwärtsrichtung"   ⇒   subscript "R"$)$  is denoted in this exercise  by  $L_{\rm R}(y)$:

$$L_{\rm R}(y) = L(x\hspace{0.05cm}|\hspace{0.05cm}y) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0)\hspace{0.05cm}|\hspace{0.05cm}y)}{{\rm Pr}(x = 1)\hspace{0.05cm}|\hspace{0.05cm}y)} \hspace{0.05cm} .$$

Hints:

• The exercise belongs to the chapter  "Soft–in Soft–out Decoder".

• Reference is made in particular to the section  "Reliability Information – Log Likelihood Ratio".

• In the last subtasks you have to clarify whether the found relations between  $L_{\rm A}, \ L_{\rm V}$  and  $L_{\rm R}$  can also be transferred to the  "2-on-$M$  channel".

• For this purpose,  we choose a bipolar approach for the input symbols:   "$0$"  →  "$+1$"  and  "$1$"   →  "$–1$".

Questions

1

How are the conditional probabilities of two random variables  $A$  and  $B$  related?

	${\rm Pr}(A\hspace{0.05cm}|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm} A)$,
	${\rm Pr}(A\hspace{0.05cm}|\hspace{0.05cm}B) = {\rm Pr}(B\hspace{0.05cm}|\hspace{0.05cm} A) \cdot {\rm Pr}(B) / {\rm Pr}(A)$,
	${\rm Pr}(A\hspace{0.05cm}|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm}A) \cdot {\rm Pr}(A) / {\rm Pr}(B)$.

2

Which equation holds for the binary channel with probabilities  ${\rm Pr}(A) = {\rm Pr}(x = 0)$  and  ${\rm Pr}(B) = {\rm Pr}(y = 0)$?

	${\rm Pr}(x = 0 | y = 0) = {\rm Pr}(y = 0 | x = 0) \cdot {\rm Pr}(x = 0) / {\rm Pr}(y = 0)$,
	${\rm Pr}(x = 0 | y = 0) = {\rm Pr}(y = 0 | x = 0) \cdot {\rm Pr}(y = 0) / {\rm Pr}(x = 0)$.

3

Under what conditions does the inference log likelihood ratio hold for all possible output values  $y ∈ \{0, \, 1\}$:     $L(x\hspace{0.05cm}|\hspace{0.05cm}y) = L(y\hspace{0.05cm}|\hspace{0.05cm}x)$   resp.   $L_{\rm R}(y) = L_{\rm V}(y)$?

	For any input distribution  ${\rm Pr}(x = 0), \ {\rm Pr}(x = 1)$.
	For the uniform distribution only:  $\hspace{0.2cm} {\rm Pr}(x = 0) = {\rm Pr}(x = 1) = 1/2$.

4

Let the initial symbol be  $y = 1$.  What inference LLR is obtained with the falsification probability  $\varepsilon = 0.1$  for equally probable symbols?

$L_{\rm R}(y = 1) = L(x | y = 1) \ = \ $

5

Let the initial symbol now be  $y = 0$. What inference log likelihood ratio is obtained for  ${\rm Pr}(x = 0) = 0.2$  and  $\varepsilon = 0.1$?

$L_{\rm R}(y = 0) = L(x | y = 0) \ = \ $

6

Can the result derived in subtask  (3)   ⇒   $L_{\rm R} = L_{\rm V} + L_{\rm A}$  also be applied to the  "2-on-M"  channel?

	Yes.
	No.

7

Can the context be applied to the AWGN channel as well?

	Yes.
	No.

Solution

Solution

(1)  For the conditional probabilities,  according to the  "Bayes' theorem"  with intersection  $A ∩ B$:

$${\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm} A) = \frac{{\rm Pr}(A \cap B)}{{\rm Pr}(A)}\hspace{0.05cm}, \hspace{0.3cm} {\rm Pr}(A \hspace{0.05cm}|\hspace{0.05cm} B) = \frac{{\rm Pr}(A \cap B)}{{\rm Pr}(B)}\hspace{0.3cm} \Rightarrow \hspace{0.3cm}{\rm Pr}(A \hspace{0.05cm}|\hspace{0.05cm} B) = {\rm Pr}(B \hspace{0.05cm}|\hspace{0.05cm} A) \cdot \frac{{\rm Pr}(A)}{{\rm Pr}(B)}\hspace{0.05cm}.$$

• Correct is the  proposition 3. 

• In the special case  ${\rm Pr}(B) = {\rm Pr}(A)$   ⇒   also the suggestion 1 would be correct.

(2)  With  $A$   ⇒   "$x = 0$"  and  $B$   ⇒   "$y = 0$"  we immediately get the equation according to  proposition 1:

$${\rm Pr}(x = 0\hspace{0.05cm}|\hspace{0.05cm} y = 0) = {\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm} x = 0) \cdot \frac{{\rm Pr}(x = 0)}{{\rm Pr}(y = 0)}\hspace{0.05cm}.$$

(3)  We compute the L–values of the inference probabilities.  Assuming  $y = 0$  holds:

$$L_{\rm R}(y= 0) \hspace{-0.15cm} \ = \ \hspace{-0.15cm} L(x\hspace{0.05cm}|\hspace{0.05cm}y= 0)= {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x = 0\hspace{0.05cm}|\hspace{0.05cm}y=0)}{{\rm Pr}(x = 1\hspace{0.05cm}|\hspace{0.05cm}y=0)} = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x=0) \cdot {\rm Pr}(x = 0) / {\rm Pr}(y = 0)}{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x = 1)\cdot {\rm Pr}(x = 1) / {\rm Pr}(y = 0)} $$

$$\Rightarrow \hspace{0.3cm} L_{\rm R}(y= 0)= {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x=0) }{{\rm Pr}(y = 0\hspace{0.05cm}|\hspace{0.05cm}x = 1)} + {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(x=0) }{{\rm Pr}(x = 1)}$$

$$\Rightarrow \hspace{0.3cm} L_{\rm R}(y= 0) = L(x\hspace{0.05cm}|\hspace{0.05cm}y= 0) = L_{\rm V}(y= 0) + L_{\rm A}(x)\hspace{0.05cm}.$$

• Similarly,  assuming  $y = 1$,  the result is:

$$L_{\rm R}(y= 1) = L(x\hspace{0.05cm}|\hspace{0.05cm}y= 1) = L_{\rm V}(y= 1) + L_{\rm A}(x)\hspace{0.05cm}.$$

• The two results can be summarized using  $y ∈ \{0, \, 1\}$  and

• the input log likelihood ratio,

$$L_{\rm A}(x) = {\rm ln} \hspace{0.15cm} \frac{ {\rm Pr}(x=0) }{ {\rm Pr}(x = 1)}\hspace{0.05cm},$$

• as well as the forward log likelihood ratio,

$$L_{\rm V}(y) = L(y\hspace{0.05cm}|\hspace{0.05cm}x) = {\rm ln} \hspace{0.15cm} \frac{{\rm Pr}(y \hspace{0.05cm}|\hspace{0.05cm}x=0) }{{\rm Pr}(y \hspace{0.05cm}|\hspace{0.05cm}x = 1)} \hspace{0.05cm},$$

as follows:

$$L_{\rm R}(y) = L(x\hspace{0.05cm}|\hspace{0.05cm}y) = L_{\rm V}(y) + L_{\rm A}(x)\hspace{0.05cm}.$$

• The identity   $L_{\rm R}(y) ≡ L_{\rm V}(y)$   requires $L_{\rm A}(x) = 0$   ⇒   equally probable symbols   ⇒   proposition 2.

(4)  From the exercise description,  you can see that with falsification probability  $\varepsilon = 0.1$,  the initial value  $y = 1$  leads to forward log likelihood ratio  $L_{\rm V}(y = 1) = \, –2.197$.

• Because of  ${\rm Pr}(x = 0) = 1/2 \ \Rightarrow \ L_{\rm A}(x) = 0$:

$$L_{\rm R}(y = 1) = L_{\rm V}(y = 1) \hspace{0.15cm}\underline{= -2.197}\hspace{0.05cm}.$$

(5)  With the same falsification probability  $\varepsilon = 0.1$   ⇒   $L_{\rm V}(y = 0)$  differs from  $L_{\rm V}(y = 1)$  only by the sign.

• With  ${\rm Pr}(x = 0) = 0.2 \ \Rightarrow \ L_{\rm A}(x) = \, -1.382$  we thus obtain:

$$L_{\rm R}(y = 0) = (+)2.197 - 1.382 \hspace{0.15cm}\underline{=+0.815}\hspace{0.05cm}.$$

(6)  The relation   $L_{\rm R} = L_{\rm V} + L_{\rm A}$   also holds for the  "2-on-$M$ channel",  regardless of the set size  $M$  of the output alphabet   ⇒   Answer Yes.

(7)  The AWGN channel is described by the outlined  "2–on–$M$–channel"  with  $M → ∞$  also   ⇒   Answer Yes.