Digital Signal Transmission/Error Probability with Intersymbol Interference

1. Digital Signal Transmission
2. Intersymbol Interfering and Equalization Methods
3. Error Probability with Intersymbol Interference

• Previous page
• Next page

• Previous page
• Next page

Gaussian receiver filter

We start from the block diagram sketched below.  The following configuration is assumed for quantitative consideration of  "intersymbol interference": 

• Rectangular NRZ basic transmission pulse  $g_s(t)$  with height  $s_0$  and duration  $T$,
  

• Gaussian-shaped receiver filter  $H_{\rm G}(f)$  with cutoff frequency $f_{\rm G}$:

$$H_{\rm E}(f) = H_{\rm G}(f) = {\rm exp}\left [- \frac{\pi \cdot f^2}{(2f_{\rm G})^2} \right ] \hspace{0.2cm} \bullet\!\!-\!\!\!-\!\!\!-\!\!\circ \hspace{0.2cm}h_{\rm E}(t) = h_{\rm G}(t) = {\rm exp}\left [- \pi \cdot (2 f_{\rm G} t)^2\right ] \hspace{0.05cm},\hspace{0.5cm}\text{note: }\hspace{0.2cm}{\rm exp} [x] = {\rm e}^x.$$

• AWGN channel   ⇒   channel frequency response  $H_{\rm K}(f) = 1 $  and  noise power-spectral density  ${\it \Phi}_n(f) = N_0/2$.

Note: 

1. We restrict ourselves in this chapter exclusively to  redundancy-free binary bipolar transmission. 
2. The ISI influence in multi-level and/or coded transmission will not be discussed until the chapter  "Intersymbol Interference for Multi-Level Transmission".

Based on the assumptions made here,  the following holds for the basic detection pulse:

$$g_d(t) = g_s(t) \star h_{\rm G}(t) = 2 f_{\rm G} \cdot s_0 \cdot \int_{t-T/2}^{t+T/2} {\rm e}^{- \pi \hspace{0.05cm}\cdot\hspace{0.05cm} (2 \hspace{0.05cm}\cdot\hspace{0.05cm} f_{\rm G}\hspace{0.05cm}\cdot\hspace{0.05cm} \tau )^2} \,{\rm d} \tau \hspace{0.05cm}.$$

The integration leads to the following equivalent results:

$$g_d(t) = s_0 \cdot \big [ {\rm Q} \left ( 2 \cdot \sqrt {2 \pi} \cdot f_{\rm G}\cdot ( t - {T}/{2})\right )- {\rm Q} \left ( 2 \cdot \sqrt {2 \pi} \cdot f_{\rm G}\cdot ( t + {T}/{2} )\right ) \big ],$$

$$g_d(t) = s_0 \cdot\big [ {\rm erfc} \left ( 2 \cdot \sqrt {\pi} \cdot f_{\rm G}\cdot ( t - {T}/{2})\right )- {\rm erfc} \left ( 2 \cdot \sqrt {\pi} \cdot f_{\rm G}\cdot ( t + {T}/{2} )\right ) \big ]\hspace{0.05cm}.$$

Block diagram for the chapter  "Error Probability with Intersymbol Interference"

Here,  two variants of the complementary Gaussian error function are used,  viz.

$${\rm Q} (x) = \frac{\rm 1}{\sqrt{\rm 2\pi}}\int_{\it x}^{+\infty}\rm e^{\it -u^{\rm 2}/\rm 2}\,d {\it u} \hspace{0.05cm},$$

$$ {\rm erfc} (\it x) = \frac{\rm 2}{\sqrt{\rm \pi}}\int_{\it x}^{+\infty}\rm e^{\it -u^{\rm 2}}\,d \it u \hspace{0.05cm}.$$

⇒   The  "Complementary Gaussian Error Functions"  provides the numerical values of the functions  ${\rm Q} (x)$  and  $0.5 \cdot {\rm erfc} (x)$.

The noise power at the output of the Gaussian receiver filter  $H_{\rm G}(f)$  is

$$\sigma_d^2 = \frac{N_0}{2} \cdot \int_{-\infty}^{+\infty} |H_{\rm G}(f)|^2 \,{\rm d} f = \frac{N_0\cdot f_{\rm G}}{\sqrt{2}}\hspace{0.05cm}.$$

$\text{From these equations one can already see:}$ 

1. The smaller the cutoff frequency  $f_{\rm G}$  of the Gaussian low-pass filter,  the smaller the noise rms value  $\sigma_d$  and consequently the better the noise performance.
   
2. However,  a small cutoff frequency leads to a strong deviation of the basic detection pulse  $g_d(t)$  from the rectangular form and thus to intersymbol interference.

$\text{Example 1:}$  The left graph shows the basic detection pulse  $g_d(t)$  at the output of a Gaussian low-pass filter  $H_{\rm G}(f)$  with the cutoff frequency  $f_{\rm G}$  when an NRZ rectangular pulse (blue curve) is applied at the input.

Basic detection pulse and noise power-spectral density  $\rm (PSD)$  with Gaussian receiver filter

One can see from this plot:

• The Gaussian low-pass filter  $H_{\rm G}(f)$  causes the detection pulse  $g_d(t)$  to be reduced and broadened compared to the transmitted pulse  $g_s(t)$  ⇒   "'time dispersion".
  

• The pulse deformation is the stronger,  the smaller the cutoff frequency  $f_{\rm G}$  is.  For example,  with  $f_{\rm G} \cdot T = 0.4$  (red curve)  the pulse maximum is already reduced to  $\approx 68\%$. 
  

• In the limiting case  $f_{\rm G} \to \infty$  the Gaussian low-pass has no effect   ⇒   $g_d(t) = g_s(t)$.  However,  in this case,  there is no noise limitation at all,  as can be seen from the right figure.

$\text{Example 2:}$  The same preconditions apply as for the last example.  The graph shows the detection signal  $d(t)$  after the Gaussian low-pass  $($before the decision$)$  for two different cutoff frequencies,  namely  $f_{\rm G} \cdot T = 0.8$  and  $f_{\rm G} \cdot T = 0.4$.  We want to analyze these images in terms of intersymbol interference.

Detection signal with Gaussian receiver filter Benedikt

In both diagrams are shown:

• the component  $d_{\rm S}(\nu \cdot T)$  of the detection signal without considering the noise  $($blue circles at the detection times$)$,
  
• the total detection signal  $d(t)$  including the noise component (yellow curve),
  
• the transmitted signal  $s(t)$  as reference signal (green dotted in the upper graph; equally valid for the lower graph).
  
  

By comparing these images, the following statements can be verified in terms of Intersymbol Interference  $\rm (ISI)$:

• With the cutoff frequency  $f_{\rm G} \cdot T = 0.8$  (upper graph),  only minor ISI result at the detection times  $($at multiples of  $T)$.  Due to the Gaussian low-pass here primarily the corners of the transmitted signal  $s(t)$  are rounded.
  
• In contrast,  in the lower image  $(f_{\rm G} \cdot T = 0.4)$  the ISI effects are clearly visible.  At the detection times  $(\nu \cdot T)$,  the  $($blue$)$  signal component  $d_{\rm S}(\nu \cdot T)$  of the detection signal can assume six different values  $($compare grid lines drawn$)$.
  
• The noise component  $d_{\rm N}(t)$ – recognizable as the difference between yellow curve and blue circles – is on average larger with $f_{\rm G} \cdot T = 0.8$  than with $f_{\rm G} \cdot T = 0.4$.
  
• This result can be explained by the right graph of  $\text{Example 1}$,  which shows the PSD of the noise component  $d_{\rm N}(t)$: 

$${\it \Phi}_{d{\rm N} }(f) = {N_0}/{2} \cdot \vert H_{\rm G}(f) \vert^2 = {N_0}/{2} \cdot {\rm exp}\left [- \frac{2\pi f^2}{(2f_{\rm G})^2} \right ] .$$

• The integral over  ${\it \Phi}_{d{\rm N} }(f)$  – i.e. the noise power  $\sigma_d^2$  – is twice as large for  $f_{\rm G} \cdot T = 0.8$  (purple curve) than with  $f_{\rm G} \cdot T = 0.4$  (red curve).

Definition and statements of the eye diagram

The above mentioned facts can also be explained by the eye diagram.

$\text{Definition:}$  The  eye diagram  (or  "eye pattern")  is the sum of all superimposed sections of the detection signal  $d(t)$,  whose duration is an integer multiple of the symbol duration  $T$.  This diagram has a certain resemblance to an eye, which led to its naming.

$\text{Example 3:}$  We assume a redundancy-free binary bipolar NRZ rectangular signal  $s(t)$  and the Gaussian low-pass filter with cutoff frequency  $f_{\rm G} \cdot T = 0.4$. 

On the left:  Eye diagram with noise  ⇒   signal  $d(t)=d_{\rm S}(t) +d_{\rm N}(t)$,
on the right:  Eye diagram without noise  ⇒   signal  $d_{\rm S}(t)$

In the graphic shown are the eye diagrams after the Gaussian low-pass,

• left inclusive the noise component   ⇒   signal  $d(t)=d_{\rm S}(t) +d_{\rm N}(t)$,
• on the right without taking noise into account   ⇒   signal  $d_{\rm S}(t)$.

This representation allows important statements about the quality of a digital transmission system:

• Only the eye diagram of the signal  $d(t)$  can be displayed metrologically on an oscilloscope,  which is triggered with the clock signal.  From this eye diagram  $($left graph$)$,  for example,  the noise rms value  $\sigma_d$  $($⇒   noise power  $\sigma_d^2)$  can be read – or rather:  estimated.
  

• The eye diagram without noise  (right graph)  refers to the signal component  $d_{\rm S}(t)$  of the detection signal and can only be determined by means of a computer simulation.  For an implemented system,  this eye diagram cannot be displayed,  since the noise component  $d_{\rm N}(t)$  cannot be eliminated.
  

• In both diagrams of this example,  $2^{11}=2048$  eye lines were drawn in each case.  In the right graph,  however,  only  $2^5 = 32$  eye lines are distinguishable because the present detection pulse  $g_d(t)$  is limited to the time range  $\vert t\vert \le 2T$ 
  $($see  graph in Example 1  with  $f_{\rm G} \cdot T = 0.4$,  red curve$)$.
  

• The inner eye lines determine the  vertical eye opening  $\ddot{o}(T_{\rm D})$.  The smaller this is,  the greater is the influence of intersymbol interference.  For a  $($ISI-free$)$  Nyquist system the vertical eye opening is maximum.  Normalized to the transmitted amplitude,  $\ddot{o}(T_{\rm D})/s_0 = 2$  is then valid.
  

• With symmetrical basic detection pulse,  the detection time  $T_{\rm D} = 0$  is optimal.  With a different value  $($for example  $T_{\rm D} = -T/10) $,   $\ddot{o}(T_{\rm D})$  would be somewhat smaller and thus the error probability would be significantly larger.  This case is indicated by the purple–dashed vertical line in the right graph.

Mean error probability

As with the previous graphs in this chapter,  we assume the following:

Eye diagram and discrete PDF of the signal component  $d_{\rm S}(t)$  from  $d(t)$

• NRZ rectangles with amplitude  $s_0$,  AWGN noise with power-spectral density  $N_0$,  where

$$10 \cdot {\rm lg}\hspace{0.1cm} \frac{s_0^2 \cdot T}{N_0}\approx 13\,{\rm dB}\hspace{0.3cm} \Rightarrow \hspace{0.3cm} \frac{N_0}{s_0^2 \cdot T} = 0.05\hspace{0.05cm}.$$

• Gaussian receiver filter with cutoff frequency  $f_{\rm G} \cdot T = 0.4$:

$$\sigma_d^2 = \frac{(N_0 /T)\cdot (f_{\rm G}\cdot T)}{\sqrt{2}}= \frac{0.05 \cdot s_0^2\cdot0.4}{\sqrt{2}}$$

$$ \Rightarrow \hspace{0.3cm} \sigma_d = \sqrt{0.0141}\cdot s_0 \approx 0.119 \cdot s_0 \hspace{0.05cm}.$$

• Let  $g_d(\nu \cdot T) \approx 0$  be valid for  $|\nu| \ge 2$.  The other basic detection pulse values are given as follows:

$$g_0 = g_d(t=0) \approx 0.68 \cdot s_0,$$

$$g_1 = g_d(t=T) \approx 0.16 \cdot s_0,$$

$$g_{-1} = g_d(t=-T) \approx 0.16 \cdot s_0\hspace{0.05cm}.$$

Let us now analyze the possible values for the signal component  $d_{\rm S}(t)$  at the detection times:

• Of the total  $32$  eye lines,  four lines intersect the ordinate  $(t = 0)$  at  $g_0 + 2 \cdot g_1 = s_0$.  These lines belong to the amplitude coefficients  "$\text{...}\hspace{0.05cm} +\hspace{-0.1cm}1,\hspace{0.05cm} {\it +\hspace{-0.05cm}1},\hspace{0.05cm} +\hspace{-0.05cm}1\hspace{0.05cm} \text{...}$".    Here,  the  "middle"  coefficient  $a_{\nu = 0}$  is highlighted in italics.
  

• The four eye lines,  each representing the coefficients  "$\text{...}\hspace{0.05cm} -\hspace{-0.1cm}1,\hspace{0.05cm} {\it +\hspace{-0.05cm}1},\hspace{0.05cm} -\hspace{-0.05cm}1,\hspace{0.05cm} \text{...}$"  result in the signal value  $d_{\rm S}(T_{\rm D} = 0) =g_0 - 2 \cdot g_1 = 0.36 \cdot s_0$.
  

• In contrast,  the signal value  $d_{\rm S}(T_{\rm D} = 0) =g_0 = 0.68 \cdot s_0$  occurs twice as often.  This goes back either to the coefficients  "$\text{...}\hspace{0.05cm} +\hspace{-0.1cm}1,\hspace{0.05cm} {\it +\hspace{-0.05cm}1},\hspace{0.05cm} -\hspace{-0.05cm}1\hspace{0.05cm} \text{...}$"  or to  "$\text{...}\hspace{0.05cm} -\hspace{-0.1cm}1,\hspace{0.05cm} {\it +\hspace{-0.05cm}1},\hspace{0.05cm} +\hspace{-0.05cm}1\hspace{0.05cm} \text{...}$". 
  

• For the  $16$  eye lines which intersect the ordinate  $T_{\rm D} = 0$  below the decision threshold  $E = 0$,  exactly mirror-image relations result.
  
  

The possible values  $d_{\rm S}(T_{\rm D})$  and their occurrence probabilities can be found in the above graph on the left side in the  (discrete)   probability density function  $\rm (PDF)$  of the noise-free detection signal samples:

$$f_{d{\rm S}}(d_{\rm S}) = {1}/{8} \cdot \delta (d_{\rm S} - s_0)+ {1}/{4} \cdot \delta (d_{\rm S} - 0.68 \cdot s_0)+ {1}/{8} \cdot \delta (d_{\rm S} - 0.36 \cdot s_0)+ $$

$$\hspace{2.15cm} + \hspace{0.2cm} {1}/{8} \cdot \delta (d_{\rm S} + s_0)+{1}/{4} \cdot \delta (d_{\rm S} + 0.68 \cdot s_0)+{1}/{8} \cdot \delta (d_{\rm S} + 0.36 \cdot s_0)\hspace{0.05cm}.$$

Thus,  the  (average)  symbol error probability of the of the ISI-afflicted system can be given.  Taking advantage of the symmetry,  one obtains with  $\sigma_d/s_0 = 0.119$:

$$p_{\rm S} = {1}/{4} \cdot {\rm Q} \left( \frac{s_0}{ \sigma_d} \right)+ {1}/{2} \cdot {\rm Q} \left( \frac{0.68 \cdot s_0}{ \sigma_d} \right)+{1}/{4} \cdot {\rm Q} \left( \frac{0.36 \cdot s_0}{ \sigma_d} \right)$$

$$\Rightarrow \hspace{0.3cm}p_{\rm S} \approx {1}/{4} \cdot {\rm Q}(8.40) +{1}/{2} \cdot {\rm Q}(5.71)+ {1}/{4} \cdot {\rm Q}(3.02)\approx {1}/{4} \cdot 2.20 \cdot 10^{-17}+ {1}/{2} \cdot 1.65 \cdot 10^{-9}+ {1}/{4} \cdot 1.26 \cdot 10^{-3} \approx 3.14 \cdot 10^{-4} \hspace{0.05cm}.$$

Note:   For redundancy-free binary bipolar transmission,  the bit error probability  $p_{\rm B}$  is identical to the symbol error probability  $p_{\rm S}$.

$\text{On the basis of this numerical example one recognizes:}$ 

1. In the presence of intersymbol interference,  the  (average)  symbol error probability  $p_{\rm S}$  is essentially determined by the inner eye lines.
   
2. The computational cost of determining  $p_{\rm S}$  can become very large,  especially if the ISI comes from very many basic detection pulse values  $g_\nu$. 

$\text{Example 4:}$ 

• If the pulse values  $g_{-5}, \text{...} \ , g_{+5}$  are different from zero and  $E \ne 0$, an averaging over  $2^{11} = 2048$  eye lines is necessary to determine the error probability  $p_{\rm S}$.

• If,  on the other hand,  only the pulse values  $g_{-1}, \ g_0, \ g_{+1}$  are different from zero and,  in addition,  the symmetry with respect to the threshold  $E = 0$  is taken into account,  the effort is reduced to averaging over four terms.

• If,  in addition,  the symmetry  $g_{-1} = g_{+1}$  applies as with the above numerical values,  then the symmetry with respect to  $T_{\rm D}$  can also be exploited and averaging over three terms is sufficient.

Worst-case error probability

In the past,  a variety of approximations for the average error probability have been given, among others: Bitte dieses Kapitel sehr kritisch checken

$\text{Definition:}$  As a very simple approximation for the actual error probability  $p_{\rm S}$,  the  worst-case error probability  (German:  "ungünstigste Fehlerwahrscheinlichkeit"   ⇒   subscript:  "U")  is often used:

"Mean symbol error probability"  $p_{\rm S}$  vs.  "worst-case symbol error probability"  $p_{\rm U}$

$$p_{\rm U} = {\rm Q} \left( \frac{\ddot{o}(T_{\rm D})/2}{ \sigma_d} \right) \hspace{0.05cm}.$$

For their calculation,  the  "worst-case symbol sequences"  are always assumed.  This means:

• The actual probability density function  $\rm (PDF)$  of the samples $d_{\rm S}(T_{\rm D})$  (left graph:  six red Dirac delta functions)  is replaced by a simplified PDF with only the inner Dirac delta functions  (right graph:  two green Diracs).
  

• For the half vertical eye opening,  with the basic detection pulse values  $g_\nu = g_d( T_{\rm D}+ \nu \cdot T)$  generally holds:

$$\ddot{o}(T_{\rm D})/{ 2}= g_0 - \sum_{\nu = 1}^{n} \vert g_{\nu} \vert- \sum_{\nu = 1}^{v} \vert g_{-\nu} \vert \hspace{0.05cm}.$$

This equation can be interpreted as follows:

• $g_0 = g_d( T_{\rm D})$  is the so-called  "main value"  of the basic detection pulse.  For Nyquist systems  $\ddot{o}(T_{\rm D})/{ 2}= g_0$  is always valid.  In the following  (mostly)   $T_{\rm D}= 0$  is set.
  

• The first sum describes the ISI influence of the  $n$  "trailers"  $($German:  "Nachläufer"  ⇒   variable  $n)$  of preceding pulses  $($sometimes we use the term  "postcursor"$)$.  Tacitly assumed is  $g_\nu = 0$  for  $\nu \gt n$. 
  

• The second sum considers the influence of the  $v$  "precursors"  $($German:  "Vorläufer"  ⇒   variable  $v)$  of following pulses under the condition  $g_{-\nu} = 0$  for  $\nu \gt v$.
  

• If all precursors and trailers are positive,  the two worst-case symbol sequences are  "$\text{...}\hspace{0.05cm} -\hspace{-0.1cm}1,\hspace{0.05cm} -\hspace{-0.05cm}1,\hspace{0.05cm} {\it +\hspace{-0.05cm}1},\hspace{0.05cm} -\hspace{-0.05cm}1,\hspace{0.05cm} -\hspace{-0.05cm}1\hspace{0.05cm} \text{...}$"  and  "$\text{...}\hspace{0.05cm} +\hspace{-0.1cm}1,\hspace{0.05cm} +\hspace{-0.05cm}1,\hspace{0.05cm} {\it -\hspace{-0.05cm}1},\hspace{0.05cm} +\hspace{-0.05cm}1,\hspace{0.05cm} +\hspace{-0.05cm}1\hspace{0.05cm} \text{...}$"  (coefficient  $a_{\nu = 0}$  is in italics in each case).  These specifications apply,  for example,  to the Gaussian receiver filter considered here.
  

• If some basic detection pulse values  $g_{\nu\ne 0}$  are negative,  this is taken into account in the above equation by the magnitude formation.  This results in other "worst–case" sequences than those just mentioned.
  

$\text{Example 5:}$  The graph shows the error probabilities of the AWGN channel as a function of the (logarithmized) quotient  $E_{\rm B}/N_0$,  namely

• the average error probability  $p_{\rm S}$  with Gaussian receiver filter  (blue circles),
  
• the worst-case error probability  $p_{\rm U}$  with Gaussian receiver filter  (blue rectangles),
  
• the smallest possible error probability according to the section "Optimal binary receiver"  (red curve).
  
  

Here,  the energy per bit is equal to  $E_{\rm B} = s_0^2 \cdot T$  (redundancy-free binary bipolar transmission,  NRZ rectangular transmitted pulses).

The left graph is for the  (normalized)  cutoff frequency  $f_{\rm G} \cdot T = 0.4$,  the right one for a broader band receiver filter with  $f_{\rm G} \cdot T = 0.8$.

Mean error probability  $p_{\rm S}$  and  worst-case error probability  $p_{\rm U}$  as a function of  $E_{\rm B}/N_0$

The results can be interpreted as follows:

• $p_{\rm U}$  is always an upper bound for the actual symbol error probability  $p_{\rm S}$.  The smaller the influence of the intersymbol interference  (large cutoff frequency),  the closer  $p_{\rm S}$  and  $p_{\rm U}$  are to each other.  For the optimal receiver  $p_{\rm S} = p_{\rm U}.$

• For a Gaussian receiver filter with  $f_{\rm G} \cdot T \ge 0.3$,  the ISI are caused by the neighboring pulses alone  $(g_2 = g_3 = \text{...} \approx 0)$,  so that a lower bound can also be given:  ${p_{\rm U} }/{ 4} \le p_{\rm S} \le p_{\rm U} \hspace{0.05cm}.$

• The strong ISI of a Gaussian receiver filter with  $f_{\rm G} \cdot T = 0.4$  leads to the fact that compared to the optimal receiver a  $6 \ \rm dB$ larger $E_{\rm B}/N_0$  must be applied (four times the power), so that the error probability does not exceed the value  $10^{-8}$. 
  

• However,  the horizontal distance between the blue  $p_{\rm S}$ curve  (marked by circles)  and the red comparison curve is not constant.  At  $p_{\rm S} = 10^{-2}$  the distance is only  $4 \ \rm dB$.
  

The right graph shows that with  $f_{\rm G} \cdot T = 0.8$  the distance to the comparison system is less than  $1 \ \rm dB$.  In the next section it is shown that with a Gaussian receiver filter the  (normalized)  cutoff frequency  $f_{\rm G} \cdot T \approx 0.8$  is the optimum.

Optimization of the cutoff frequency

For system optimization and system comparison, it turns out to be convenient,  instead of using the worst-case error probability  $p_{\rm U}$  to use the  "worst–case signal–to–noise power ratio"  $\text{(S/N ratio)}$:

SNR as a function of the cutoff frequency of a Gaussian low-pass filter

$$\rho_{\rm U} = [\ddot{o}(T_{\rm D})]^2/ \sigma_d^2.$$

• In the case of Gaussian perturbation, the following relationship exists:

$$p_{\rm U} = {\rm Q} \left( \sqrt{\rho_{\rm U}} \right) \hspace{0.05cm}.$$

• The error probability  $p_{\rm S}$  can also be formally expressed by a S/N ratio via the Q–function:

$$\rho_d = \left[{\rm Q}^{-1} \left( p_{\rm S} \right)\right]^2 \hspace{0.05cm}.$$

The diagram shows the two quantities  $\rho_d$  and  $\rho_{\rm U}$  in logarithmic form depending on the normalized cutoff frequency  $f_{\rm G} \cdot T$  of a Gaussian receiver filter,  where  $10 \cdot {\rm lg}\hspace{0.1cm} E_{\rm B}/N_0 = 13 \ \rm dB$  is the basis.

• The blue circles are for  $10 \cdot {\rm lg}\hspace{0.1cm} \rho_d$   ⇒   "mean detection SNR",
  

• The blue squares mark  $10 \cdot {\rm lg}\hspace{0.1cm} \rho_{\rm U}$   ⇒   "worst-case detection SNR".

For comparison,  the result for the  "optimal binary receiver"  is also plotted as a red horizontal line.  For this optimum binary system holds:

$$\rho_d = \rho_{\rm U} = {2 \cdot E_{\rm B}}/{ N_0}\hspace{0.3cm}\Rightarrow \hspace{0.3cm} 10 \cdot {\rm lg}\hspace{0.1cm} \rho_d = 10 \cdot {\rm lg}\hspace{0.1cm} \rho_{\rm U} \approx 16\,{\rm dB} \hspace{0.05cm}.$$

One can see from the plot:

1. The optimization criterion  $\rho_d$  leads to the optimal cutoff frequency  $f_\text{G, opt} \cdot T = 0.8$.  A smaller cutoff frequency results in stronger intersymbol interference  $($smaller eye opening$)$,  a larger cutoff frequency results in a larger noise power  $\sigma_d^2$.
   
2. Such a Gaussian receiver filter with  $f_\text{G, opt} \cdot T \approx 0.8$  leads to the signal-to-noise ratio  $10 \cdot {\rm lg}\hspace{0.1cm} \rho_d \approx 15 \ \rm dB$  and thus to the error probability  $p_{\rm S} \approx 10^{-8}$.  For comparison:   With the optimal receiver  $($impulse response matched to the transmitter$)$,  the results are  $10 \cdot {\rm lg}\hspace{0.1cm} \rho_d \approx 16 \ \rm dB$  and  $p_{\rm S} \approx 10^{-10}$.
   
3. However,  the graph also shows that the much simpler optimization criterion  $ \rho_{\rm U}$  $($or  $ p_{\rm U})$  leads approximately to the same optimal cutoff frequency  $f_\text{G, opt} \cdot T = 0.8$.  For this cutoff frequency,  we obtain the worst-case SNR  $10 \cdot {\rm lg}\hspace{0.1cm} \rho_{\rm U} \approx 14.7 \ \rm dB$  and the worst-case error probability  $p_{\rm U} \approx 3 \cdot 10^{-8}$.
   
4. If the cutoff frequency  $f_\text{G} \cdot T < 0.27$,  the vertical eye opening will always be  $\ddot{o}(T_{\rm D}) \equiv 0$.  This is called a  "closed eye".  As a consequence,  some worst-case symbol sequences would always be wrongly decided even without noise.  A systematic error occurs.
   
5. Further investigations have shown that the optimization criterion  $ \rho_{\rm U}$  is sufficient even with smaller  $E_{\rm B}/N_0$.  Thus,  for a distortion-free channel   ⇒   $H_{\rm K}(f) = 1$,  the optimal cutoff frequency of the Gaussian low-pass always results in  $f_\text{G, opt} \cdot T \approx 0.8$,  at least in a realistic approach.
   
   

⇒   All statements of this chapter can be reproduced with the interactive HTML5/JavaScript applet  "Eye diagram and eye opening".

Exercises for the chapter

Exercise 3.2: Eye Pattern according to Gaussian Low-Pass

Exercise 3.2Z: Optimum Cut-off Frequency for Gaussian Low-pass