Expected Values and Moments

Moment calculation as ensemble average

The probability density function $\rm (PDF)$, like the cumulative distribution function $\rm (CDF)$, provides very extensive information about the random variable under consideration. Less, but more compact information is provided by the so-called »expected values« and »moments«.

Their calculation possibilities have already been given for value-discrete random variables in the chapter »Moments of a Discrete Random Variable«.

Now these integrative descriptive quantities »expected value« and »moment« are considered in the context of the probability density function $\rm (PDF)$ of value-continuous random variables and thus formulated more generally.

$\text{Definitions:}$

The »expected value« with respect to any weighting function $g(x)$ can be calculated with the PDF $f_{\rm x}(x)$ in the following way:

$${\rm E}\big[g (x ) \big] = \int_{-\infty}^{+\infty} g(x)\cdot f_{x}(x) \,{\rm d}x.$$

Substituting into this equation for $g(x) = x^k$ we get the »moment of $k$-th order«:

$$m_k = {\rm E}\big[x^k \big] = \int_{-\infty}^{+\infty} x^k\cdot f_{x} (x ) \, {\rm d}x.$$

From this equation follows.

with $k = 1$ for the »first order moment« ⇒ »$($linear$)$ mean«:

$$m_1 = {\rm E}\big[x \big] = \int_{-\infty}^{ \rm +\infty} x\cdot f_{x} (x ) \,{\rm d}x,$$

with $k = 2$ for the »second order moment«:

$$m_2 = {\rm E}\big[x^{\rm 2} \big] = \int_{-\infty}^{ \rm +\infty} x^{ 2}\cdot f_{ x} (x) \,{\rm d}x.$$

For a $M$–level random variable, the formulas given here again yield the equations already given in the second chapter $($»Calculation as an ensemble average«$)$:

$$m_1 = \sum\limits_{\mu=1}^{ M}\hspace{0.15cm}p_\mu\cdot x_\mu,$$

$$m_2 = \sum\limits_{\mu= 1}^{ M}\hspace{0.15cm}p_\mu\cdot x_\mu^2.$$

Here it is taken into account that the integral over the Dirac delta function $δ(x)$ is equal to $1$. In relation to signals, the following terms are also common:

$m_1$ indicates the »DC component«.

$m_2$ corresponds to the »signal power« $($referred to the unit resistance $1 \ Ω)$.

For example, if $x$ denotes a voltage, then according to these equations $m_1$ has the unit "${\rm V}$" and $m_2$ the unit "${\rm V}^2$". If one wants to indicate the power in »Watt« $\rm (W)$, then $m_2$ must still be divided by the resistance value $R$.

Central moments

$\text{Definition:}$ Especially important in statistics are the »central moments«, which in contrast to the conventional moments are each related to the mean $m_1$:

$$\mu_k = {\rm E}\big[(x-m_{\rm 1})^k\big] = \int_{-\infty}^{+\infty} (x-m_{\rm 1})^k\cdot f_x(x) \,\rm d \it x.$$

The non–centered moments $m_k$ can be directly converted to the centered moments $\mu_k$:

$$\mu_k = \sum\limits_{\kappa= 0}^{k} \left( \begin{array}{*{2}{c} } k \\ \kappa \\ \end{array} \right)\cdot m_\kappa \cdot (-m_1)^{k-\kappa}.$$

According to the general equations of the »last section« the formal quantities $m_0 = 1$ and $\mu_0 = 1$ result.

For the first order central moment, according to the above definition always holds: $\mu_1 = 0$.

In the opposite direction, the following equations hold for $k = 1$, $k = 2$, and so on:

$$m_k = \sum\limits_{\kappa= 0}^{k} \left( \begin{array}{*{2}{c}} k \\ \kappa \\ \end{array} \right)\cdot \mu_\kappa \cdot {m_1}^{k-\kappa}.$$

$\text{Example 1:}$ All moments of a binary random variable with probabilities ${\rm Pr}(0) = 1 - p$ and ${\rm Pr}(1) = p$ are of equal value:

$$m_1 = m_2 = m_3 = m_4 = \hspace{0.05cm}\text{...} \hspace{0.05cm}= p.$$

Using the above equations, we then obtain for the first three central moments:

$$\mu_2 = m_2 - m_1^2 = p -p^2, $$

$$\mu_3 = m_3 - 3 \cdot m_2 \cdot m_1 + 2 \cdot m_1^3 = p - 3 \cdot p^2 + 2 \cdot p^3, $$

$$ \mu_4 = m_4 - 4 \cdot m_3 \cdot m_1 + 6 \cdot m_2 \cdot m_1^2 - 3 \cdot m_1^4 = p - 4 \cdot p^2 + 6 \cdot p^3- 3 \cdot p^4. $$

Some common central moments

From the »last definition« the following additional characteristics can be derived:

$\text{Definition:}$ The »variance« $σ^2$ of the considered random variable is the second order central moment ⇒ $\mu_2.$

The variance $σ^2$ corresponds physically to the »alternating current power« and $σ$ gives the standard deviation.
From the first and the second moment, the variance can be calculated according to »Steiner's theorem« in the following way:

$$\sigma^{2} = m_2 - m_1^{2}.$$

$\text{Definition:}$ The »Charlier's skewness« $S$ denotes the third central moment related to $σ^3$.

For symmetrical probability density function, this parameter is always $S=0$.
The larger $S = \mu_3/σ^3$, the more asymmetrical is the PDF around the mean $m_1$.
For example, for the »exponential distribution« the $($positive$)$ skewness $S =2$, and this is independent of the distribution parameter $λ$.
For positive skewness $(S > 0)$ one speaks of a »right–skewed« or of a »left–sloping distribution«; this slopes flatter on the right side than on the left.
When the skewness $S < 0$ there is a »left–skewed« or a »right–steep distribution«; such a distribution falls flatter on the left side than on the right.

$\text{Definition:}$ The fourth-order central moment is also used for statistical analysis; The quotient $K = \mu_4/σ^4$ is called »kurtosis«.

For a »Gaussian distributed random variable« this always yields the value $K = 3$.
Using also the so-called »excess« $\gamma = K - 3$, also known under the term »overkurtosis«.
This parameter can be used, for example, to check whether a random variable at hand is approximately Gaussian ⇒ excess $\gamma \approx 0$.

$\text{Example 2:}$

If the PDF has fewer offshoots than the Gaussian distribution, the kurtosis $K < 3$. For example, for the »uniform distribution«: $K = 1.8$ ⇒ $\gamma = - 1.2$.

In contrast, $K > 3$ indicates that the offshoots are more pronounced than for the Gaussian distribution. For example, for the »exponential distribution«: $K = 9$.

The »Laplace distribution« ⇒ »two-sided exponential distribution« results in a slightly smaller kurtosis $K = 6$ ⇒ excess $\gamma = 3$.

Moment calculation as time average

The expected value calculation according to the previous equations of this section corresponds to a »ensemble averaging«, that is, averaging over all possible values $x_\mu$.

However, the moments $m_k$ can also be determined as »time averages« if the stochastic process generating the random variable is stationary and ergodic:

The exact definition for such a stationary and ergodic random process can be found in $\text{Chapter 4.4}$.
In the following »time-averaging« is always denoted by a sweeping line.
For discrete time, the random signal $x(t)$ is replaced by the random sequence $〈x_ν〉$.
For finite sequence length, these time averages are with $ν = 1, 2,\hspace{0.05cm}\text{...}\hspace{0.05cm} , N$:

$$m_k=\overline{x_{\nu}^{k}}=\frac{1}{N} \cdot \sum\limits_{\nu=1}^{N}x_{\nu}^{k},$$

$$m_1=\overline{x_{\nu}}=\frac{1}{N} \cdot \sum\limits_{\nu=1}^{N}x_{\nu},$$

$$m_2=\overline{x_{\nu}^{2}}=\frac{1}{N} \cdot \sum\limits_{\nu=1}^{N}x_{\nu}^{2}.$$

If the moments $($or expected values$)$ are to be determined by simulation, in practice this is usually done by time averaging. The corresponding computational algorithm differs only mariginally for value-discrete and value-continuous random variables.

⇒ The topic of this chapter is illustrated with examples in the (German language) learning video
»Momentenberechnung bei diskreten Zufallsgrößen» $\Rightarrow$ »Moment calculation for discrete random variables«.

Characteristic function

$\text{Definition:}$ Another special case of an expected value is the »characteristic function«, where for the valuation function is to be set $g(x) = {\rm e}^{\hspace{0.03cm}{\rm j} \hspace{0.03cm}{\it Ω}\hspace{0.05cm}x}$:

$$C_x({\it \Omega}) = {\rm E}\big[{\rm e}^{ {\rm j} \hspace{0.05cm} {\it \Omega} \hspace{0.05cm} x}\big] = \int_{-\infty}^{+\infty} {\rm e}^{ {\rm j} \hspace{0.05cm} {\it \Omega} \hspace{0.05cm} x}\cdot f_{\rm x}(x) \hspace{0.1cm}{\rm d}x.$$

A comparison with the chapter »Fourier Transform and Inverse Fourier Transform« in the book »Signal Representation« shows that the characteristic function can be interpreted as the »inverse Fourier transform of the probability density function«:

$$C_x ({\it \Omega}) \hspace{0.3cm} \circ\!\!-\!\!\!-\!\!\!-\!\!\bullet \hspace{0.3cm} f_{x}(x).$$

If the random variable $x$ is dimensionless, then the argument $\it Ω$ of the characteristic function is also without unit.

The symbol $\it Ω$ was chosen because the argument here has some relation to the angular frequency in the second Fourier integral
$($compared to the representation in the $f$–domain, however, the factor $2\pi$ is missing in the exponent$)$.
But it is again insistently pointed out that $C_x({\it Ω})$ would correspond to the »time function« and $f_{x}(x)$ to the »spectral function«, if one wants to establish a relation to systems theory.

$\text{Calculation possibility:}$

Developing the complex function ${\rm e}^{\hspace{0.03cm}{\rm j} \hspace{0.03cm}{\it Ω}\hspace{0.05cm}x }$ into a »power series» and interchanges expectation value formation and summation, the series representation of the characteristic function follows:

$$C_x ( {\it \Omega}) = 1 + \sum_{k=1}^{\infty}\hspace{0.2cm}\frac{m_k}{k!} \cdot ({\rm j} \hspace{0.01cm}{\it \Omega})^k .$$

More properties of the characteristic function can be found in $\text{Exercise 3.4}$.

$\text{Example 3:}$ For a symmetric binary $($two–point distributed$)$ random variable $x ∈ \{\pm1\}$ with probabilities

$${\rm Pr}(-1) = {\rm Pr}(+1) = 1/2$$

the characteristic function is cosine. The analogue in systems theory is that the spectrum of a cosine signal with angular frequency ${\it Ω}_{\hspace{0.03cm}0}$ consists of two Dirac delta functions at $±{\it Ω}_{\hspace{0.03cm}0}$.

$\text{Example 4:}$ A uniform distribution between $±y_0$ has the following characteristic function according to the »Fourier transform laws«:

$$C_y({\it \Omega}) = \frac{1}{2 y_0} \cdot \int_{-y_0}^{+y_0} {\rm e}^{ {\rm j} \hspace{0.05cm} {\it \Omega} \hspace{0.05cm} y} \,{\rm d}y = \frac{ {\rm e}^{ {\rm j} \hspace{0.05cm} y_0 \hspace{0.05cm}{\it \Omega} } - {\rm e}^{ - {\rm j} \hspace{0.05cm} y_0 \hspace{0.05cm} {\it \Omega} } }{2 {\rm j} \cdot y_0 \cdot {\it \Omega} } = \frac{ {\rm sin}(y_0 \cdot {\it \Omega})}{ y_0 \cdot {\it \Omega} } = {\rm si}(y_0 \cdot {\it \Omega}). $$

We already know the function ${\rm si}(x) = \sin(x)/x = {\rm sinc}(x/\pi) $ from the book »Signal Representation«.

Exercises for the chapter

Exercise 3.3: Moments for Cosine-square PDF

Exercise 3.3Z: Moments for Triangular PDF

Exercise 3.4: Characteristic Function