# Stochastic System Theory

## # OVERVIEW OF THE FIFTH MAIN CHAPTER #

This chapter describes the influence of a filter on the  »auto-correlation function«  $\rm (ACF)$  and  »the power-spectral density  $\rm (PSD)$«  of stochastic signals.

In detail,  this chapter covers:

• the  »calculation of ACF and PSD«  at the filter output  ("Stochastic System Theory"),
• the structure and representation of  »digital filters«  (non-recursive and recursive),
• the  »dimensioning«  of the filter coefficients for a given ACF,
• the meaning of the  »matched filter«  for the SNR maximization of communication systems,
• the properties of the  »Wiener-Kolmogorow filter«  for the signal reconstruction.

## System model and problem definition

As in the book  "Linear and Time Invariant Systems",  we consider the setup sketched on the right, where the system characterized both

• by the impulse response  $h(t)$
• as well as by its frequency response  $H(f)$

is described unambiguously.  The relationship between these descriptive quantities in the time and frequency domain is given by the  $\text{Fourier transformation}$ Filter influence on  "spectrum"  and
"power-spectral density"

If we apply the signal  $x(t)$  to the input and denote the output signal by  $y(t)$,  the classical system theory provides the following statements:

• The output signal  $y(t)$  results from the  $\text{convolution}$  between the input signal  $x(t)$  and the impulse response  $h(t)$.  The following equation is equally valid for deterministic and stochastic signals:
$$y(t) = x(t) \ast h(t) = \int_{-\infty}^{+\infty} x(\tau)\cdot h ( t - \tau) \,\,{\rm d}\tau.$$
• For deterministic signals,  one usually takes a roundabout route using the spectral functions.  The spectrum  $X(f)$  is the Fourier transform of  $x(t)$.  The multiplication with the frequency response  $H(f)$  leads to the output spectrum  $Y(f)$.  From this,  the signal  $y(t)$  can be obtained by the Fourier inverse transformation.
• In the case of stochastic signals this procedure fails,  because then the time functions  $x(t)$  and  $y(t)$  are not predictable for all times  from ­$–∞$  to  $+∞$  and thus,  the corresponding amplitude spectra  $X(f)$  and  $Y(f)$  do not exist at all.  In this case,  we have to switch to the  $\text{power-spectral density}$  defined in the last chapter.

## Amplitude spectrum and power-spectral density

We consider an ergodic random process  $\{x(t)\}$,  whose auto-correlation function  $φ_x(τ)$  is assumed to be known.  The power-spectral density  ${\it Φ}_x(f)$  is then also uniquely determined via the Fourier transform and the following statements hold:

1. The  $\text{power-spectral density}$  ${\it Φ}_x(f)$  can be given – as well as the auto-correlation function  $φ_x(τ)$ – for each individual pattern function of the stationary and ergodic random process  $\{x(t)\}$,  even if the specific course of  $x(t)$  is explicitly unknown.

2. The  $\text{amplitude spectrum}$  $X(f)$,  on the other hand,  is undefined because if the spectral function  $X(f)$  is known, the entire time function  $x(t)$  from  $–∞$  to  $+∞$  would also have to be known via the Fourier inverse transform,  which clearly cannot be the case for a stochastic signal.

3. If a time section of finite time duration  $T_{\rm M}$  is known according to the sketch on the right,  the Fourier transform can of course be applied to it again.

$\text{Theorem:}$  The following relationship exists between the power-spectral density  ${\it Φ}_x(f)$  of the infinite time random signal  $x(t)$  and the amplitude spectrum  $X_{\rm T}(f)$  of the finite time section  $x_{\rm T}(t)$:

$${ {\it \Phi}_x(f)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot \vert X_{\rm T}(f)\vert ^2.$$

$\text{Proof:}$  Previously,  the  $\text{auto-correlation function}$  of an ergodic process with the random signal  $x(t)$  was given as follows:

$${ {\it \varphi}_x(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot\int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x(t)\cdot x(t + \tau)\hspace{0.1cm} \rm d \it t.$$
• It is permissible to replace the function  $x(t)$,  which is unbounded in time, by the function  $x_{\rm T}(t)$,  which is bounded on the time range  $-T_{\rm M}/2$  to  $+T_{\rm M}/2$.    $x_{\rm T}(t)$  corresponds to the spectrum  $X_{\rm T}(f)$,  and by applying the  $\text{first Fourier integral}$  and the  $\text{shifting theorem}$:
$${ {\it \varphi}_x(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot \int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x_{\rm T}(t)\cdot \int^{+\infty}_{-\infty}X_{\rm T}(f)\cdot {\rm e}^{ {\rm j}2 \pi f ( t + \tau) } \hspace{0.1cm} \rm d \it f \hspace{0.1cm} \rm d \it t.$$
• After splitting the exponent and swapping the time and frequency integrals,  we get:
$${ {\it \varphi}_x(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot \int^{+\infty}_{-\infty}X_{\rm T}(f)\cdot \left[ \int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x_{\rm T}(t)\cdot {\rm e}^{ {\rm j}2 \pi f t } \hspace{0.1cm} \rm d \it t \right] \cdot {\rm e}^{ {\rm j}2 \pi f \tau} \hspace{0.1cm} \rm d \it f.$$
• The inner integral describes the conjugate-complex spectrum  ${X_{\rm T} }^{\star}(f)$.  It further follows that:
$${ {\it \varphi}_x(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot \int^{+\infty}_{-\infty}\vert X_{\rm T}(f)\vert^2 \cdot {\rm e}^{ {\rm j}2 \pi f \tau} \hspace{0.1cm} \rm d \it f.$$
$${ {\it \varphi}_x(\tau)} = \int^{+\infty}_{-\infty}{\it \Phi}_x(f) \cdot {\rm e}^{ {\rm j}2 \pi f \tau} \hspace{0.1cm} \rm d \it f ,$$
shows the validity of the above relation:
$${ {\it \Phi}_x(f)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} }\cdot \vert X_{\rm T}(f)\vert^2.$$
q.e.d.

## Power-spectral density of the filter output signal

Combining the statements made in the last two sections,  we arrive at the following important result:

$\text{Theorem:}$  The power-spectral density  $\rm (PSD)$  at the output of a linear time-invariant system with frequency response  $H(f)$  is obtained as the product

• of the  "input power-spectral density"  ${\it Φ}_x(f)$
• and the  "power transfer function"  $\vert H(f)\vert ^2$.
$${ {\it \Phi}_y(f)} = { {\it \Phi}_x(f)} \cdot \vert H(f)\vert ^2.$$

$\text{Proof:}$  Starting from the three relations already derived before:

$${ {\it \Phi}_x(f)} =\hspace{-0.1cm} \lim_{T_{\rm M}\to\infty}\hspace{0.01cm} \frac{1}{ T_{\rm M} }\hspace{-0.05cm}\cdot\hspace{-0.05cm} \vert X_{\rm T}(f)\vert^2,$$
$${ {\it \Phi}_y(f)} =\hspace{-0.1cm} \lim_{T_{\rm M}\to\infty}\hspace{0.01cm} \frac{1}{ T_{\rm M} }\hspace{-0.05cm}\cdot\hspace{-0.05cm}\vert Y_{\rm T}(f)\vert^2,$$
$$Y_{\rm T}(f) = X_{\rm T}(f) \hspace{-0.05cm}\cdot\hspace{-0.05cm} H(f).$$

Substituting these equations into each other,  we get the above result.

q.e.d.

The following example illustrates the relationship with white noise.

$\text{Example 1:}$  At the input of a Gaussian low-pass filter with the frequency response

$$H(f) = {\rm e}^{- \pi \hspace{0.03cm}\cdot \hspace{0.03cm}(f/\Delta f)^2}$$

white noise  $x(t)$  with noise power density  ${ {\it \Phi}_x(f)} =N_0/2$  is present   ⇒   two-sided representation.  Then, the following holds for the power-spectral density of the output signal:

$${ {\it \Phi}_y(f)} = \frac {N_0}{2} \cdot {\rm e}^{- 2 \pi \hspace{0.03cm}\cdot \hspace{0.03cm}(f/\Delta f)^2}.$$

The diagram shows the signals and power-spectral densities at the filter input and output.

Notes:

1. The signal  $x(t)$  – strictly speaking – cannot be plotted at all,  since it has an infinite power   ⇒   integral over  ${\it Φ}_x(f)$  from  $-\infty$  to  $+\infty$.
2. The output signal  $y(t)$  has a lower frequency than  $x(t)$  and a finite power corresponding to the integral over  ${\it Φ}_y(f)$.
3. In one-sided representation,  (only) for  $f>0$  would hold:  ${ {\it \Phi}_x(f)} =N_0$.  The statements  (1)  and  (2)  would also apply here in the same way.

## The auto-correlation function of the filter output signal

The calculated power-spectral density  $\rm (PSD)$  can also be written as follows:

$${{\it \Phi}_y(f)} = {{\it \Phi}_x(f)} \cdot H(f) \cdot H^{\star}(f).$$

$\text{Theorem:}$  The corresponding auto-correlation function  $\rm (ACF)$  is then obtained according to the  $\text{Fourier transform laws}$  and by applying the  $\text{convolution theorem}$:

$${ {\it \varphi}_y(\tau)} = { {\it \varphi}_x(\tau)} \ast h(\tau)\ast h(- \tau).$$

In the transition from the spectral to the time domain, note:

• The Fourier retransforms are to be inserted in each case, namely
$${{\it \varphi}_y(\tau)} \circ\hspace{0.05cm}\!\!\!-\!\!\!-\!\!\!-\!\!\bullet\,{{\it \Phi}_y(f)}, \hspace{0.5cm}{{\it \varphi}_x(\tau)} \circ\hspace{0.05cm}\!\!\!-\!\!\!-\!\!\!-\!\!\!\bullet\,{{\it \Phi}_x(f)}, \hspace{0.5cm}{h(\tau)} \circ\hspace{0.05cm}\!\!\!-\!\!\!-\!\!\!-\!\!\bullet\,{H(f)}, \hspace{0.5cm}{h(-\tau)} \circ\hspace{0.05cm}\!\!\!-\!\!\!-\!\!\!-\!\!\!\bullet\,{H^{\star}(f)}.$$
• Moreover,  each multiplication becomes a convolution operation.

$\text{Example 2:}$  We consider again the same scenario as  in  $\text{Example 1}$,  but this time in the time domain.  It holds:

• Two-sided white noise power density:  ${ {\it \Phi}_x(f)} =N_0/2$,
• Gaussian filter:   $H(f) = {\rm e}^{- \pi \hspace{0.03cm}\cdot \hspace{0.03cm}(f/\Delta f)^2}\hspace{0.3cm}\Rightarrow \hspace{0.3cm} h(t) = \Delta f \cdot {\rm e}^{- \pi \hspace{0.03cm}\cdot \hspace{0.03cm}(\Delta f \hspace{0.03cm}\cdot \hspace{0.03cm}t)^2}.$

One can see from this diagram:

1. The ACF of the input signal is now a Dirac delta function with weight  $N_0/2$.
2. By convolution twice with the  (here also Gaussian)  impulse response  $h(t)$  or  $h(–t)$  one obtains the ACF  $φ_y(τ)$  of the output signal.
3. Thus,  the ACF  $φ_y(τ)$  of the output signal is also Gaussian.
4. The ACF value at  $τ = 0$  is identical to the area of the power-spectral density  ${\it Φ}_y(f)$  and characterizes the signal power (variance)  $σ_y^2$.
5. In contrast, the area at  $φ_y(τ)$  gives the PSD value:  ${\it Φ}_y(f = \rm 0)=N_0/2$.

## Cross-correlation function between input and output signal

We again consider a filter with the frequency response  $H(f)$  and the impulse response  $h(t)$.  Further applies:

1. The stochastic input signal  $x(t)$  is a sample function of the ergodic random process  $\{x(t)\}$.

2. The corresponding auto-correlation function  $\rm (ACF)$  at the filter input is thus  $φ_x(τ)$,  while the power-spectral density  $\rm (PSD)$  is denoted by  ${\it Φ}_x(f)$.

3. The corresponding descriptors of the ergodic random process  $\{y(t)\}$  at the filter output are
• the random output signal  $y(t)$,
• the auto-correlation function  $φ_y(τ)$
• and the conductance power-spectral density  ${\it Φ}_y(f)$.

$\text{Theorem:}$  For the  »cross-correlation function«  $\rm (CCF)$  between the input  and the output signal holds:

$${ {\it \varphi}_{xy}(\tau)} = h(\tau)\ast { {\it \varphi}_x(\tau)} .$$

Here,  $h(τ)$  denotes the impulse response of the filter  $($with the time variable  $τ$  instead of  $t)$  and  ${ {\it \varphi}_{x}(\tau)}$  denotes the ACF of the input signal.

$\text{Proof:}$  In general,  for the cross-correlation function between two signals  $x(t)$  and  $y(t)$:

$${ {\it \varphi}_{xy}(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm}\frac{1}{ T_{\rm M} }\cdot\int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x(t)\cdot y(t + \tau)\hspace{0.1cm} \rm d \it t.$$
• With the generally valid relation  $y(t) = h(t) \ast x(t)$  and the formal integration variable  $θ$,  we can also write for this:
$${ {\it \varphi}_{xy}(\tau)} = \lim_{T_{\rm M}\to\infty}\hspace{0.2cm}\frac{1}{ T_{\rm M} }\cdot\int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x(t)\cdot \int^{+\infty}_{-\infty} h(\theta) \cdot x(t + \tau - \theta)\hspace{0.1cm}{\rm d}\theta\hspace{0.1cm}{\rm d} \it t.$$
• By interchanging the two integrals and subtracting the limit into the integral, we obtain:
$${ {\it \varphi}_{xy}(\tau)} = \int^{+\infty}_{-\infty} h(\theta) \cdot \left[ \lim_{T_{\rm M}\to\infty}\hspace{0.2cm} \frac{1}{ T_{\rm M} } \cdot\int^{+T_{\rm M}/2}_{-T_{\rm M}/2}x(t)\cdot x(t + \tau - \theta)\hspace{0.1cm} \hspace{0.1cm} {\rm d} t \right]{\rm d}\theta.$$
• The expression in the square brackets gives the ACF value at the input at time  $τ - θ$:
$${ {\it \varphi}_{xy}(\tau)} = \int^{+\infty}_{-\infty}h(\theta) \cdot \varphi_x(\tau - \theta)\hspace{0.1cm}\hspace{0.1cm} {\rm d}\theta = h(\tau)\ast { {\it \varphi}_x(\tau)} .$$
• However,  the remaining integral describes the convolution operation in detailed notation.
q.e.d.

$\text{Conclusion:}$  In the frequency domain,  the corresponding equation is:

$${ {\it \Phi}_{xy}(f)} = H(f)\cdot{ {\it \Phi}_x(f)} \hspace{0.3cm} \Rightarrow \hspace{0.3cm} H(f) = \frac{ {\it \Phi}_{xy}(f)}{ {\it \Phi}_{x}(f)}.$$

This equation shows that the filter frequency response  $H(f)$  from a measurement with stochastic excitation can be calculated completely  – i.e., both magnitude and phase –  if the following descriptive quantities are determined:

• the statistical characteristics at the input, either the ACF  $φ_x(τ)$  or the   PSD ${\it Φ}_x(f)$,
• as well as the cross-correlation function  $φ_{xy}(τ)$  or its Fourier transform  ${\it Φ}_{xy}(f)$.