Information Theory/Differential Entropy - Revision history

Guenter at 13:29, 28 February 2023

2023-02-28T13:29:30Z

Guenter at 11:18, 24 February 2023

2023-02-24T11:18:28Z

Guenter at 11:15, 24 February 2023

2023-02-24T11:15:24Z

Guenter at 11:13, 24 February 2023

2023-02-24T11:13:11Z

Show changes

Hwang at 10:38, 22 February 2023

2023-02-22T10:38:38Z

Hwang at 10:36, 22 February 2023

2023-02-22T10:36:26Z

Hwang at 10:34, 22 February 2023

2023-02-22T10:34:16Z

Hwang at 10:31, 22 February 2023

2023-02-22T10:31:27Z

Hwang at 10:30, 22 February 2023

2023-02-22T10:30:00Z

Hwang at 10:28, 22 February 2023

2023-02-22T10:28:45Z

@@ Line 3: / Line 3: @@
 |Untermenü=Information Theory for Continuous Random Variables
 |Vorherige Seite=Anwendung auf die Digitalsignalübertragung
-|Nächste Seite=AWGN–Kanalkapazität bei wertkontinuierlichem Eingang
+|Nächste Seite=AWGN_Channel_Capacity_for_Continuous-Valued_Input
 }}

@@ Line 430: / Line 430: @@
-If we only allow mean-free random variables, we circumvent the problem.&nbsp; Thus the&nbsp; [https://en.wikipedia.org/wiki/Lagrange_multiplier $&raquo;\text{Lagrange multiplier}$&laquo;]:
+If we only allow mean-free random variables, we circumvent the problem.&nbsp; Thus the&nbsp; [https://en.wikipedia.org/wiki/Lagrange_multiplier "Lagrange multiplier"]:
 :$$L= \hspace{0.1cm}  \hspace{0.05cm} \int_{-\infty}^{+\infty} \hspace{-0.1cm}  f_X(x) \cdot {\rm log} \hspace{0.1cm} \frac{1}{ f_X(x) } \hspace{0.1cm}{\rm d}x \hspace{0.1cm}+ \hspace{0.1cm}
@@ Line 439: / Line 439: @@
 \int_{-\infty}^{+\infty}\hspace{-0.1cm}  x^2 \cdot f_X(x)  \hspace{0.1cm}{\rm d}x \hspace{0.05cm}.$$
-Following a similar procedure as in the&nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_peak_constraint|$\text{proof of the peak constraint}$]]&nbsp; it turns out, that the "best possible" function must be  &nbsp; $g_X(x) \sim {\rm e}^{–λ_2\hspace{0.05cm} · \hspace{0.05cm} x^2}$&nbsp; &nbsp;  ⇒  &nbsp;   [[Theory_of_Stochastic_Signals/Gaussian_Distributed_Random_Variables|$\text{Gaussian distribution}$]]:
+Following a similar procedure as in the&nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_peak_constraint|"proof of the peak constraint"]]&nbsp; it turns out, that the "best possible" function must be  &nbsp; $g_X(x) \sim {\rm e}^{–λ_2\hspace{0.05cm} · \hspace{0.05cm} x^2}$&nbsp; &nbsp;  ⇒  &nbsp;   [[Theory_of_Stochastic_Signals/Gaussian_Distributed_Random_Variables|"Gaussian distribution"]]:
 :$$g_X(x) ={1}/{\sqrt{2\pi  \sigma^2}} \cdot {\rm e}^{
 - \hspace{0.05cm}{x^2}/{(2 \sigma^2)} }\hspace{0.05cm}.$$
-However, we use here for the explicit proof the&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Informational_divergence_-_Kullback-Leibler_distance|$\text{Kullback–Leibler distance}$]]&nbsp; between a suitable general PDF&nbsp; $f_X(x)$&nbsp; and the Gaussian PDF&nbsp; $g_X(x)$:
+However, we use here for the explicit proof the&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Informational_divergence_-_Kullback-Leibler_distance|"Kullback–Leibler distance"]]&nbsp; between a suitable general PDF&nbsp; $f_X(x)$&nbsp; and the Gaussian PDF&nbsp; $g_X(x)$:
 :$$D(f_X \hspace{0.05cm} ||  \hspace{0.05cm}g_X) = \int_{-\infty}^{+\infty} \hspace{0.02cm}

@@ Line 424: / Line 424: @@
 <br>
 Let's start by explaining the term:
-*Actually,&nbsp; it is not the power &nbsp; ⇒ the&nbsp; [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments|"second moment"]]&nbsp; $m_2$ that  is limited,&nbsp; but the&nbsp; [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments|"second central moment"]]&nbsp;  ⇒ &nbsp; variance&nbsp; $μ_2 = σ^2$.
+*Actually,&nbsp; it is not the power &nbsp; ⇒ the&nbsp; [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Moment_calculation_as_ensemble_average|"second moment"]]&nbsp; $m_2$ that  is limited,&nbsp; but the&nbsp; [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments|"second central moment"]]&nbsp;  ⇒ &nbsp; variance&nbsp; $μ_2 = σ^2$.
 *We are now looking for the maximum differential entropy under the constraint&nbsp; ${\rm E}\big[|X - m_1|^2 \big] ≤ σ^2$.

@@ Line 338: / Line 338: @@
 [[File:EN_Inf_T_4_1_S5a_v5.png|right|frame|Differential entropy of power-constrained random variables]]
 *the&nbsp; [[Theory_of_Stochastic_Signals/Gaußverteilte_Zufallsgrößen|$\text{Gaussian distribution}$]]'',
-*the&nbsp; [[Theory_of_Stochastic_Signals/Exponentialverteilte_Zufallsgrößen#Zweiseitige_Exponentialverteilung_.E2.80.93_Laplaceverteilung|$\text{Laplace distribution}$]]&nbsp;  ⇒  &nbsp; a two-sided exponential distribution,
+*the&nbsp; [[Theory_of_Stochastic_Signals/Exponentially_Distributed_Random_Variables#Two-sided_exponential_distribution_-_Laplace_distribution|$\text{Laplace distribution}$]]&nbsp;  ⇒  &nbsp; a two-sided exponential distribution,
 *the  (one-sided) &nbsp;  [[Theory_of_Stochastic_Signals/Exponentialverteilte_Zufallsgrößen#One-sided_exponential_distribution|$\text{exponential distribution}$]].

@@ Line 320: / Line 320: @@
 :$$h_{\rm max}(X) = {\rm log} \hspace{0.1cm} (2A)\hspace{0.05cm}.$$
 Here, the appropriate parameter&nbsp; ${\it \Gamma}_{\rm A} = 2$&nbsp; is maximal.
-You will find the&nbsp; [[Information_Theory/Differentielle_Entropie#Beweis:_Maximale_differentielle_Entropie_bei_Spitzenwertbegrenzung|$\text{proof}$]]&nbsp; at the end of this chapter.}}
+You will find the&nbsp; [[Information_Theory/Differential_Entropy#Proof:_Maximum_differential_entropy_with_peak_constraint|$\text{proof}$]]&nbsp; at the end of this chapter.}}
@@ Line 357: / Line 357: @@
 leads to the maximum differential entropy,&nbsp; independent of the mean&nbsp; $m_1$:
 :$$h(X) = 1/2 \cdot {\rm log} \hspace{0.1cm} (2\pi{\rm e} \cdot \sigma^2)\hspace{0.3cm}\Rightarrow\hspace{0.3cm}{\it \Gamma}_{\rm L} < 2π{\rm e} ≈ 17.08\hspace{0.05cm}.$$
-You will find the&nbsp; [[Information_Theory/Differentielle_Entropie#Beweis:_Maximale_differentielle_Entropie_bei_Leistungsbegrenzung|$\text{proof}$]]&nbsp; at the end of this chapter.}}
+You will find the&nbsp; [[Information_Theory/Differential_Entropy#Proof:_Maximum_differential_entropy_with_power_constraint|$\text{proof}$]]&nbsp; at the end of this chapter.}}

@@ Line 220: / Line 220: @@
 A pseudo-unit must be added in each case:
-*„nat” when using&nbsp; "ln"  &nbsp; ⇒ &nbsp;  natural logarithm,
+*"nat" when using&nbsp; "ln"  &nbsp; ⇒ &nbsp;  natural logarithm,
-*„bit” when using&nbsp; "log<sub>2</sub>"  &nbsp; ⇒ &nbsp;  binary logarithm.}}
+*"bit" when using&nbsp; "log<sub>2</sub>"  &nbsp; ⇒ &nbsp;  binary logarithm.}}
-While the (conventional) entropy of a discrete random variable&nbsp; $X$&nbsp; is always&nbsp; $H(X) ≥ 0$&nbsp;, the differential entropy&nbsp; $h(X)$&nbsp; of a continuous random variable can also be negative.&nbsp; From this it is already evident that&nbsp; $h(X)$&nbsp;  in contrast to&nbsp; $H(X)$&nbsp; cannot be interpreted as „uncertainty”.
+While the (conventional) entropy of a discrete random variable&nbsp; $X$&nbsp; is always&nbsp; $H(X) ≥ 0$&nbsp;, the differential entropy&nbsp; $h(X)$&nbsp; of a continuous random variable can also be negative.&nbsp; From this it is already evident that&nbsp; $h(X)$&nbsp;  in contrast to&nbsp; $H(X)$&nbsp; cannot be interpreted as "uncertainty".
 [[File:P_ID2854__Inf_T_4_1_S3a_neu.png|right|frame|PDF of an uniform distributed random variable]]
@@ Line 234: / Line 234: @@
 {\rm ln} \hspace{0.1cm} \big[ {x_{\rm max}\hspace{-0.05cm} - \hspace{-0.05cm}x_{\rm min} }\big ]  \cdot \big [ \frac{1}{x_{\rm max}\hspace{-0.05cm} - \hspace{-0.05cm}x_{\rm min} } \big ]_{x_{\rm min} }^{x_{\rm max} }={\rm ln} \hspace{0.1cm} \big[ {x_{\rm max}\hspace{-0.05cm} - \hspace{-0.05cm}x_{\rm min} } \big]\hspace{0.05cm}.\end{align*} $$
-The equation for the differential entropy in „bit” is: &nbsp;
+The equation for the differential entropy in "bit" is: &nbsp;
 :$$h(X) = \log_2 \big[x_{\rm max} – x_{ \rm min} \big].$$
@@ Line 347: / Line 347: @@
 The result differs only by the pseudo-unit
-*„nat” when using&nbsp; $\ln$&nbsp; or
+*"nat" when using&nbsp; $\ln$&nbsp; or
-*„bit” when using&nbsp; $\log_2$.
+*"bit" when using&nbsp; $\log_2$.
 <br clear=all>
 {{BlaueBox|TEXT=
@@ Line 420: / Line 420: @@
 \int_{-\infty}^{+\infty}\hspace{-0.1cm}  x^2 \cdot f_X(x)  \hspace{0.1cm}{\rm d}x \hspace{0.05cm}.$$
-Following a similar procedure as in the&nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_peak_constraint|$\text{proof of the peak constraint}$]]&nbsp; it turns out, that the „best possible” function must be  &nbsp; $g_X(x) \sim {\rm e}^{–λ_2\hspace{0.05cm} · \hspace{0.05cm} x^2}$&nbsp; &nbsp;  ⇒  &nbsp;   [[Theory_of_Stochastic_Signals/Gaussian_Distributed_Random_Variables|$\text{Gaussian distribution}$]]:
+Following a similar procedure as in the&nbsp; [[Information_Theory/Differentielle_Entropie#Proof:_Maximum_differential_entropy_with_peak_constraint|$\text{proof of the peak constraint}$]]&nbsp; it turns out, that the "best possible" function must be  &nbsp; $g_X(x) \sim {\rm e}^{–λ_2\hspace{0.05cm} · \hspace{0.05cm} x^2}$&nbsp; &nbsp;  ⇒  &nbsp;   [[Theory_of_Stochastic_Signals/Gaussian_Distributed_Random_Variables|$\text{Gaussian distribution}$]]:
 :$$g_X(x) ={1}/{\sqrt{2\pi  \sigma^2}} \cdot {\rm e}^{

@@ Line 195: / Line 195: @@
 {{GraueBox|TEXT=
-$\text{Example 3:}$&nbsp; As in&nbsp; $\text{example 2}$&nbsp;, we consider a asymmetrical triangular PDF &nbsp; $($between&nbsp; $0$&nbsp; and&nbsp; $1)$.&nbsp; Its differential entropy, as calculated in&nbsp; [[Aufgaben:4.2_Dreieckförmige_WDF|"Exercise 4.2"]]&nbsp; results in&nbsp;
+$\text{Example 3:}$&nbsp; As in&nbsp; $\text{example 2}$&nbsp;, we consider a asymmetrical triangular PDF &nbsp; $($between&nbsp; $0$&nbsp; and&nbsp; $1)$.&nbsp; Its differential entropy, as calculated in&nbsp; [[Aufgaben:Exercise_4.2:_Triangular_PDF|"Exercise 4.2"]]&nbsp; results in&nbsp;
 [[File:EN_Inf_T_4_1_S2c.png|right|frame|Entropy of the asymmetrical triangular PDF after quantization ]]
 :$$h(X) = \hspace{0.05cm}-0.279 \ \rm bit.$$

← Older revision		Revision as of 10:30, 22 February 2023
Line 87:		Line 87:


−	We ~~apologise~~ for this formal inaccuracy.}}	+	We apologize for this formal inaccuracy.}}

@@ Line 29: / Line 29: @@
 ==Properties of continuous random variables==
 <br>
-Up to now,&nbsp; "discrete random variables"&nbsp; of the form&nbsp; $X = \{x_1,\ x_2, \hspace{0.05cm}\text{...}\hspace{0.05cm} , x_μ, \text{...} ,\ x_M\}$&nbsp; have always been considered, which from an information-theoretical point of view are completely characterised by their&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Probability_mass_function_and_probability_density_function|$\text{probability mass function}$]]&nbsp; $\rm (PMF)$:
+Up to now,&nbsp; "discrete random variables"&nbsp; of the form&nbsp; $X = \{x_1,\ x_2, \hspace{0.05cm}\text{...}\hspace{0.05cm} , x_μ, \text{...} ,\ x_M\}$&nbsp; have always been considered, which from an information-theoretical point of view are completely characterized by their&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Probability_mass_function_and_probability_density_function|$\text{probability mass function}$]]&nbsp; $\rm (PMF)$:
 :$$P_X(X) = \big [ \hspace{0.1cm}
@@ Line 73: / Line 73: @@
 We use in this chapter for a&nbsp; &raquo;'''probability density function'''&laquo;&nbsp; $\rm (PDF)$&nbsp;   the representation form&nbsp; $f_X(x)$&nbsp; often used in the literature, where holds:
 *$X$&nbsp; denotes the (discrete or continuous) random variable,
-*$x$&nbsp; is a possible realisation of&nbsp; $X$ &nbsp; ⇒ &nbsp; $x ∈ X$.
+*$x$&nbsp; is a possible realization of&nbsp; $X$ &nbsp; ⇒ &nbsp; $x ∈ X$.
@@ Line 83: / Line 83: @@
 In other&nbsp; $\rm LNTwww$ books, we often write so as not to use up two characters for one variable:
-*For the PDF&nbsp; $f_x(x)$   &nbsp; ⇒ &nbsp; no distinction between random variable and realisation.
+*For the PDF&nbsp; $f_x(x)$   &nbsp; ⇒ &nbsp; no distinction between random variable and realization.
 *For the CDF&nbsp; $F_x(r) = {\rm Pr}(x ≤ r)$ &nbsp; ⇒ &nbsp; here one needs a second variable in any case.
@@ Line 401: / Line 401: @@
 \hspace{0.05cm}.$$
-Any other random variable with the PDF property &nbsp;$f_X(\vert x \vert  > A) = 0$ &nbsp; leads to a smaller differential entropy, characterised by the parameter &nbsp;${\it \Gamma}_{\rm A} < 2$.}}
+Any other random variable with the PDF property &nbsp;$f_X(\vert x \vert  > A) = 0$ &nbsp; leads to a smaller differential entropy, characterized by the parameter &nbsp;${\it \Gamma}_{\rm A} < 2$.}}
 ==Proof: Maximum differential entropy with power constraint==
@@ Line 458: / Line 458: @@
   {1}/{2} \cdot {\rm log} \hspace{0.1cm} (2\pi{\rm e} \cdot \sigma^2) \hspace{0.5cm} \Rightarrow\hspace{0.5cm} {\it \Gamma}_{\rm L} = 2\pi{\rm e}
 \hspace{0.05cm}.$$
-Any other  continuous random variable&nbsp; $X$&nbsp; with variance&nbsp; ${\rm E}\big[ \vert X – m_1 \vert ^2 \big] ≤ σ^2$&nbsp; leads to a smaller value,&nbsp; characterised by the parameter ${\it \Gamma}_{\rm L}  < 2πe$. }}
+Any other  continuous random variable&nbsp; $X$&nbsp; with variance&nbsp; ${\rm E}\big[ \vert X – m_1 \vert ^2 \big] ≤ σ^2$&nbsp; leads to a smaller value,&nbsp; characterized by the parameter ${\it \Gamma}_{\rm L}  < 2πe$. }}