Exercise 3.5: Kullback-Leibler Distance and Binomial Distribution - Revision history

Hwang at 12:19, 18 January 2023

2023-01-18T12:19:21Z

Hwang at 23:48, 12 November 2022

2022-11-12T23:48:42Z

Noah at 08:13, 24 September 2021

2021-09-24T08:13:58Z

Guenter at 12:33, 31 August 2021

2021-08-31T12:33:53Z

Guenter at 12:16, 31 August 2021

2021-08-31T12:16:56Z

Guenter at 12:15, 31 August 2021

2021-08-31T12:15:29Z

Guenter at 12:08, 31 August 2021

2021-08-31T12:08:16Z

Guenter at 15:35, 30 August 2021

2021-08-30T15:35:27Z

Noah at 21:19, 21 August 2021

2021-08-21T21:19:31Z

Noah at 21:18, 21 August 2021

2021-08-21T21:18:31Z

Show changes

@@ Line 165: / Line 165: @@
 * With &nbsp;$&lambda; = 1$&nbsp;, the&nbsp; <u>linear means</u>&nbsp; of the two random variables coincide:
 :$$m_X = m_Y= 1.$$
-* With &nbsp;$&lambda; = 0.9$ the&nbsp; <u>quadratic means</u>&nbsp; agree:
+* With &nbsp;$&lambda; = 0.9$ the&nbsp; <u>second moments</u>&nbsp; agree:
 :$$m_X + \sigma_X^2 = m_Y + \sigma_Y^2= 1.8.$$

@@ Line 46: / Line 46: @@
 Hints:
 *The exercise belongs to the chapter&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables|Some preliminary remarks on two-dimensional random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Informational_divergence_-_Kullback-Leibler_distance|Relative entropy &ndash; Kullback-Leibler distance]].
+*In particular, reference is made to the section&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Informational_divergence_-_Kullback-Leibler_distance|Relative entropy &ndash; Kullback-Leibler distance]].
 *To keep the numerical calculations within limits, the following auxiliary quantities are given;&nbsp;  here&nbsp; $\rm \lg$&nbsp; denotes the logarithm to base&nbsp; $10$:
 :$$A\hspace{0.05cm}' =

← Older revision		Revision as of 08:13, 24 September 2021
Line 1:		Line 1:

−	{{quiz-Header\|Buchseite=~~Informationstheorie~~/~~Einige Vorbemerkungen zu zweidimensionalen Zufallsgrößen~~	+	{{quiz-Header\|Buchseite=Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables
	}}		}}

@@ Line 111: / Line 111: @@
 ===Solution===
 {{ML-Kopf}}
-'''(1)'''&nbsp; guarantees the best approximation&nbsp; ${\rm Pr}(X > I) = 0$ &nbsp; &#8658; &nbsp; $\underline{I = 5}$.&nbsp; Thus, for the probability that&nbsp; $X =I = 5$&nbsp;, we get:
+'''(1)'''&nbsp; With the binomial distribution, all probabilities are&nbsp; ${\rm Pr}(X > I) = 0$ &nbsp; &#8658; &nbsp; $\underline{I = 5}$.&nbsp; Thus, for the probability that&nbsp; $X =I = 5$,&nbsp; we get:
 :$${\rm Pr} (X = 5) = {5 \choose 5} \cdot p^{5} =  p^{5}  \approx 0.0003 \hspace{0.05cm}.$$
 Thus one obtains for
@@ Line 125: / Line 125: @@
 *Using&nbsp; $D(P_Y \hspace{0.05cm}|| \hspace{0.05cm} P_X)$&nbsp; would always result in an infinite value regardless of&nbsp; $&lambda;$&nbsp; since for&nbsp; $\mu &#8805; 6$&nbsp;:
 :$$P_X (X = \mu) = 0 \hspace{0.05cm},\hspace{0.3cm}P_Y (Y = \mu) \ne 0 \hspace{0.05cm}.$$
-*Even though the probabilities&nbsp; $P_Y (Y = \mu)$&nbsp; become very small for large&nbsp; $&mu;$&nbsp; they are still 'infinitely larger' than&nbsp; $P_X (X = \mu)$.
+*Even though the probabilities&nbsp; $P_Y (Y = \mu)$&nbsp; become very small for large&nbsp; $&mu;$&nbsp; they are still&nbsp; "infinitely larger"&nbsp; than&nbsp; $P_X (X = \mu)$.
@@ Line 132: / Line 132: @@
 '''(3)'''&nbsp; We use the first Kullback&ndash;Leibler distance:
 :$$D = D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y) =\hspace{0.2cm} \sum_{\mu = 0}^{5}  P_X(\mu) \cdot {\rm log}_2 \hspace{0.1cm} \frac{P_X(\mu)}{P_Y(\mu)} \hspace{0.05cm}.$$
-*Using the logarithm of base ten&nbsp; $(\lg)$&nbsp, we obtain for the Poisson approximation with &nbsp;$\lambda = 1$:
+*Using the logarithm of base ten&nbsp; $(\lg)$,&nbsp; we obtain for the Poisson approximation with &nbsp;$\lambda = 1$:
 :$$D \hspace{0.05cm}' = 0.3277 \cdot {\rm lg} \hspace{0.1cm} \frac{0.3277}{0.3679} + A \hspace{0.05cm}' =
 -0.016468 + 0.021944 = 0.005476 \hspace{0.05cm}.$$
@@ Line 142: / Line 142: @@
 :$$H\hspace{0.05cm}'(Y) = -{\rm E} \left [{\rm lg} \hspace{0.1cm} {P_Y(Y)} \right ]
 = -2 \cdot 0.3679 \cdot {\rm lg} \hspace{0.1cm} (0.3679) - B\hspace{0.05cm}' = 0.31954 + 0.24717 = 0.56126.$$
-*Converting to 'bit' gives the result we are looking for:
+*Converting to&nbsp; "bit"&nbsp; gives the result we are looking for:
 :$$H(Y) = \frac{0.56126}{{\rm lg} \hspace{0.1cm}(2)}
 \hspace{0.15cm} \underline {= 1.864\ {\rm (bit)}} \hspace{0.05cm}.$$
@@ Line 149: / Line 149: @@
 '''(5)'''&nbsp; Correct is <u>statement 1</u>.&nbsp; In the numerical calculation of the Kullback&ndash;Leibler distance
-* the contribution of the&nbsp; $&mu;$&ndash;th term is positive if&nbsp; $P_Y(\mu) > P_X(\mu)$,
+* the contribution of the&nbsp; $&mu;$&ndash;th term is positive,&nbsp; if&nbsp; $P_Y(\mu) > P_X(\mu)$,
-* the contribution of the&nbsp; $&mu;$&ndash;th term is negative if&nbsp; $P_Y(\mu) < P_X(\mu)$.
+* the contribution of the&nbsp; $&mu;$&ndash;th term is negative,&nbsp; if&nbsp; $P_Y(\mu) < P_X(\mu)$.
@@ Line 168: / Line 168: @@
 :$$m_X + \sigma_X^2 = m_Y + \sigma_Y^2= 1.8.$$
-Whether this statement is relevant, we leave undecided.&nbsp; Because: &nbsp; Due to the steady increase of &nbsp; $H(Y)$&nbsp; with increasing&nbsp; $&lambda;$&nbsp;, it is clear that for some&nbsp; $&lambda;$&ndash;value &nbsp; $H(Y) = H(X)$&nbsp; must indeed hold.
+Whether this statement is relevant, we leave undecided.&nbsp;
 {{ML-Fuß}}

@@ Line 46: / Line 46: @@
 Hints:
 *The exercise belongs to the chapter&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables|Some preliminary remarks on two-dimensional random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Relative_Entropie_.E2.80.93_Kullback.E2.80.93Leibler.E2.80.93Distanz|Relative entropy &ndash; Kullback-Leibler distance]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Informational_divergence_-_Kullback-Leibler_distance|Relative entropy &ndash; Kullback-Leibler distance]].
 *To keep the numerical calculations within limits, the following auxiliary quantities are given;&nbsp;  here&nbsp; $\rm \lg$&nbsp; denotes the logarithm to base&nbsp; $10$:
 :$$A\hspace{0.05cm}' =

@@ Line 24: / Line 24: @@
 \hspace{0.05cm},\hspace{0.15cm}  \text{...}\hspace{0.1cm} ,\hspace{0.15cm} {\mu}\hspace{0.05cm}, \hspace{0.05cm}\text{...}\hspace{0.1cm}\}\hspace{0.05cm},$$
 * Probabilites:
-:$$P_Y (Y = \mu) = \frac{\lambda^{\mu}}{\mu !} \cdot {\rm e}^{\lambda} \hspace{0.05cm},$$
+:$$P_Y (Y = \mu) = \frac{\lambda^{\mu}}{\mu !} \cdot {\rm e}^{-\lambda} \hspace{0.05cm},$$
 * Expected values:
 :$$m_Y = \sigma_Y^2 = \lambda\hspace{0.05cm}.$$
@@ Line 34: / Line 34: @@
 [[File:EN_Inf_A_3_4_B.png|right|frame|Enclosed results table]]
 :$$D(P_Y \hspace{0.05cm}|| \hspace{0.05cm} P_X) \hspace{0.15cm}  =  \hspace{0.15cm}  {\rm E} \left [ {\rm log}_2 \hspace{0.1cm} \frac{P_Y(X)}{P_X(X)}\right ] \hspace{0.2cm}=\hspace{0.2cm} \sum_{\mu = 0}^{\infty}  P_Y(\mu) \cdot {\rm log}_2 \hspace{0.1cm} \frac{P_Y(\mu)}{P_X(\mu)} \hspace{0.05cm}.$$
-If&nbsp; $\log_2$&nbsp; is used, the pseudo&ndash;unit 'bit' must be added to the numerical value.
+If&nbsp; $\log_2$&nbsp; is used, the pseudo&ndash;unit&nbsp; "bit"&nbsp; must be added to the numerical value.
 In the adjacent table, the Kullback&ndash;Leibler&ndash;distance&nbsp;  $D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y)$&nbsp;   (in&nbsp; "bit")&nbsp; between the binomial PMF&nbsp;  $P_X(\cdot)$&nbsp;  and some Poisson approximations&nbsp;  $P_Y(\cdot)$&nbsp;    $($with five different rates $\lambda)$&nbsp; is entered.
 *The respective entropy &nbsp;$H(Y)$, which also depends on the rate &nbsp;$\lambda$,&nbsp; is given in the first row.
 *The columns for&nbsp; $\lambda = 1$&nbsp; are to be completed in subtasks&nbsp; '''(3)'''&nbsp; and '''(4)'''&nbsp;.
 *In subtask&nbsp; '''(6)'''&nbsp; these results are to be interpreted.
@@ Line 85: / Line 84: @@
-{Calculate the appropriate Kullback&ndash;Leibler distance&nbsp;  $($abbreviated here as&nbsp; $D$&nbsp; $)$&nbsp; for &nbsp;$\lambda = 1$.&nbsp; Consider the auxiliary quantity &nbsp;$A\hspace{0.05cm}'$.
+{Calculate the appropriate Kullback&ndash;Leibler distance&nbsp;  $($abbreviated here as&nbsp; $D)$&nbsp;  for &nbsp;$\lambda = 1$.&nbsp; Consider the auxiliary quantity &nbsp;$A\hspace{0.05cm}'$.
 |type="{}"}
 $D \ = \ $ { 0.0182 3% } $\ \rm bit$
@@ Line 97: / Line 96: @@
 {Which of the following statements are true?
 |type="[]"}
-+ In the &nbsp;$H(Y)$ calculation, all terms have the same sign.
++ In the &nbsp;$H(Y)$&nbsp; calculation, all terms have the same sign.
-- In the &nbsp; $D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y)$ calculation all terms have the same sign.
+- In the &nbsp; $D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y)$&nbsp; calculation all terms have the same sign.

@@ Line 4: / Line 4: @@
 [[File:EN_Inf_A_3_4_A.png|right|frame|Given probabilities]]
-We assume here the&nbsp; [[Theory_of_Stochastic_Signals/Binomialverteilung|binomial distribution]]&nbsp;, which is characterised by the parameters &nbsp;$I$&nbsp; and &nbsp;$p$&nbsp; &nbsp; <br>&#8658; &nbsp; see the book "Stochastische Signaltheorie":
+We assume here the&nbsp; [[Theory_of_Stochastic_Signals/Binomialverteilung|binomial distribution]],&nbsp; which is characterised by the parameters &nbsp;$I$&nbsp; and &nbsp;$p$&nbsp; &nbsp; <br>&#8658; &nbsp; see the book&nbsp; [[Theory_of_Stochastic_Signals|"Theory of Stochastic Signals"]]:
 * Range of Values:
@@ Line 15: / Line 15: @@
 * Variance:
 :$$\sigma_X^2 = I  \cdot p \cdot (1-p)\hspace{0.05cm}.$$
-In the part of the table highlighted in red, the probabilities&nbsp; $P_X(X = \mu$)&nbsp; of the binomial distribution under consideration are given.&nbsp; In subtask&nbsp; '''(1)'''&nbsp; you are to determine the corresponding distribution parameters &nbsp;$I$&nbsp; and &nbsp;$p$&nbsp;.
+In the part of the table highlighted in red, the probabilities&nbsp; $P_X(X = \mu$)&nbsp; of the binomial distribution under consideration are given.&nbsp; In subtask&nbsp; '''(1)'''&nbsp; you are to determine the corresponding distribution parameters &nbsp;$I$&nbsp; and &nbsp;$p$.
-This given binomial distribution is to be approximated here by a&nbsp; [[Theory_of_Stochastic_Signals/Poissonverteilung|Poisson distribution]]&nbsp; $Y$&nbsp;, characterised by the rate &nbsp;$\lambda$:
+This given binomial distribution is to be approximated here by a&nbsp; [[Theory_of_Stochastic_Signals/Poissonverteilung|Poisson distribution]]&nbsp; $Y$,&nbsp; characterised by the rate &nbsp;$\lambda$:
 * Range of values:
@@ Line 28: / Line 28: @@
 :$$m_Y = \sigma_Y^2 = \lambda\hspace{0.05cm}.$$
-In order to assess whether the probability function&nbsp; $P_X(X)$&nbsp; is sufficiently well approximated by&nbsp; $P_Y(Y)$&nbsp;, one can resort to the so-called&nbsp; <i>Kullback&ndash;Leibler distances</i>&nbsp; ('''KLD''')&nbsp; , sometimes also called&nbsp;  "relative entropies"&nbsp; in the literature.
+In order to assess whether the probability mass function&nbsp; $P_X(X)$&nbsp; is sufficiently well approximated by&nbsp; $P_Y(Y)$,&nbsp; one can resort to the so-called&nbsp; <b>Kullback&ndash;Leibler distances</b>&nbsp; $\rm (KLD)$,&nbsp; sometimes also called&nbsp;  "relative entropies"&nbsp; in the literature.
 Adapted to the present example, these are:
@@ Line 36: / Line 36: @@
 If&nbsp; $\log_2$&nbsp; is used, the pseudo&ndash;unit 'bit' must be added to the numerical value.
-In the adjacent table, the Kullback&ndash;Leibler&ndash;distance&nbsp;  $D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y)$&nbsp;   (in 'bit') between the binomial PMF&nbsp;  $P_X(\cdot)$&nbsp;  and some Poisson approximations&nbsp;  $P_Y(\cdot)$&nbsp;    $($with five different rates $\lambda)$&nbsp; is entered.
+In the adjacent table, the Kullback&ndash;Leibler&ndash;distance&nbsp;  $D(P_X \hspace{0.05cm}|| \hspace{0.05cm} P_Y)$&nbsp;   (in&nbsp; "bit")&nbsp; between the binomial PMF&nbsp;  $P_X(\cdot)$&nbsp;  and some Poisson approximations&nbsp;  $P_Y(\cdot)$&nbsp;    $($with five different rates $\lambda)$&nbsp; is entered.
-*The respective entropy &nbsp;$H(Y)$, which also depends on the rate &nbsp;$\lambda$&nbsp; , is given in the first row.
+*The respective entropy &nbsp;$H(Y)$, which also depends on the rate &nbsp;$\lambda$,&nbsp; is given in the first row.
 *The columns for&nbsp; $\lambda = 1$&nbsp; are to be completed in subtasks&nbsp; '''(3)'''&nbsp; and '''(4)'''&nbsp;.
@@ Line 46: / Line 46: @@
 Hints:
-*The task belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on 2D random variables]].
+*The exercise belongs to the chapter&nbsp; [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables|Some preliminary remarks on two-dimensional random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Relative_Entropie_.E2.80.93_Kullback.E2.80.93Leibler.E2.80.93Distanz|Relative entropy &ndash; Kullback-Leibler Distance]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Relative_Entropie_.E2.80.93_Kullback.E2.80.93Leibler.E2.80.93Distanz|Relative entropy &ndash; Kullback-Leibler distance]].
 *To keep the numerical calculations within limits, the following auxiliary quantities are given;&nbsp;  here&nbsp; $\rm \lg$&nbsp; denotes the logarithm to base&nbsp; $10$:
-:$$A' =
+:$$A\hspace{0.05cm}' =
 .4096 \cdot {\rm lg} \hspace{0.1cm} \frac{0.4096}{0.3679} +
 .2048 \cdot {\rm lg} \hspace{0.1cm} \frac{0.2048}{0.1839} +
@@ Line 56: / Line 55: @@
 .0064 \cdot {\rm lg} \hspace{0.1cm} \frac{0.0064}{0.0153} +
 .0003 \cdot {\rm lg} \hspace{0.1cm} \frac{0.0003}{0.0031} \hspace{0.05cm},$$
-:$$B' =
+:$$B\hspace{0.05cm}' =
 .1839 \cdot {\rm lg} \hspace{0.1cm} (0.1839) +
 .0613 \cdot {\rm lg} \hspace{0.1cm} (0.0613) +
@@ Line 63: / Line 62: @@
 .0005 \cdot {\rm lg} \hspace{0.1cm} (0.0005) +
 .0001 \cdot {\rm lg} \hspace{0.1cm} (0.0001)$$
-:$$\Rightarrow \hspace{0.3cm} A'  \hspace{0.15cm} \underline {= 0.021944}  \hspace{0.05cm},\hspace{0.5cm}
+:$$\Rightarrow \hspace{0.3cm} A\hspace{0.05cm}'  \hspace{0.15cm} \underline {= 0.021944}  \hspace{0.05cm},\hspace{0.5cm}
-B'  \hspace{0.15cm} \underline {= -0.24717}  \hspace{0.05cm}.$$
+B\hspace{0.05cm}'  \hspace{0.15cm} \underline {= -0.24717}  \hspace{0.05cm}.$$
@@ Line 70: / Line 69: @@
 <quiz display=simple>
-{What are the characteristics of the present binomial distribution? &nbsp; <i>Hint:</i> Enter (at most) one decimal place.
+{What are the characteristics of the present binomial distribution? &nbsp; Hint:&nbsp; Enter (at most) one decimal place.
 |type="{}"}
 $I \hspace{0.47cm}  = \ $ { 5 3% }
@@ Line 104: / Line 103: @@
 {How do you interpret the completed results table?
 |type="[]"}
-+ According Kullback&ndash;Leibler distance, you should choose &nbsp;$\lambda = 1$&nbsp;.
++ According Kullback&ndash;Leibler distance, you should choose &nbsp;$\lambda = 1$.
 - $\lambda = 1$&nbsp; guarantees the best approximation &nbsp;$H(Y) &asymp; H(X)$.

@@ Line 47: / Line 47: @@
 Hints:
 *The task belongs to the chapter&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen|Some preliminary remarks on 2D random variables]].
-*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Relative_Entropie_.E2.80.93_Kullback.E2.80.93Leibler.E2.80.93Distanz|Relative Entropie &ndash; Kullback-Leibler Distance]].
+*In particular, reference is made to the page&nbsp; [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Relative_Entropie_.E2.80.93_Kullback.E2.80.93Leibler.E2.80.93Distanz|Relative entropy &ndash; Kullback-Leibler Distance]].
 *To keep the numerical calculations within limits, the following auxiliary quantities are given;&nbsp;  here&nbsp; $\rm \lg$&nbsp; denotes the logarithm to base&nbsp; $10$:
@@ Line 155: / Line 155: @@
-[[File:P_ID2761__Inf_A_3_4_C.png|right|frame|Kullback–Leibler distance and Entropie]]
+[[File:P_ID2761__Inf_A_3_4_C.png|right|frame|Kullback–Leibler distance and entropy]]