Aufgaben:Exercise 4.4: Conventional Entropy and Differential Entropy - Revision history

Maintenance script: Fix interlanguage link: resolve redirect chain

2026-03-16T15:57:03Z

Fix interlanguage link: resolve redirect chain

Maintenance script: Join split $$ formula lines for correct rendering

2026-03-16T13:48:37Z

Join split $$ formula lines for correct rendering

Maintenance script: Add German interlanguage link

2026-03-16T12:32:52Z

Add German interlanguage link

← Older revision		Revision as of 14:32, 16 March 2026
Line 128:		Line 128:

	[[Category:Information Theory: Exercises\|^4.1 Differential Entropy^]]		[[Category:Information Theory: Exercises\|^4.1 Differential Entropy^]]
			[[de:Aufgaben:4.04_Zusammenhang_h(X)_und_H(Z)]]

Hwang at 12:44, 17 November 2022

2022-11-17T12:44:23Z

← Older revision		Revision as of 14:44, 17 November 2022
Line 16:		Line 16:
	The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.		The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.

	From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the ~~page~~  [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .		From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the section  [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .

	In the section  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:		In the section  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:
Line 33:		Line 33:
	Hints:		Hints:
	*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].		*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].
	*Useful hints for solving this task can be found in particular on the ~~page~~  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of continuous random variables after quantization]].		*Useful hints for solving this task can be found in particular in the section  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of continuous random variables after quantization]].


Line 72:		Line 72:
	===Solution===		===Solution===
	{{ML-Kopf}}		{{ML-Kopf}}
	'''(1)'''  According to the corresponding theory ~~page~~, with  $x_{\rm min} = 0$  and  $x_{\rm max} = 1/2$:		'''(1)'''  According to the corresponding theory section, with  $x_{\rm min} = 0$  and  $x_{\rm max} = 1/2$:
	:$$h(X) = {\rm log}_2 \hspace{0.1cm} (x_{\rm max} - x_{\rm min}) = {\rm log}_2 \hspace{0.1cm} (1/2) \hspace{0.15cm}\underline{= - 1\,{\rm bit}}\hspace{0.05cm}.$$		:$$h(X) = {\rm log}_2 \hspace{0.1cm} (x_{\rm max} - x_{\rm min}) = {\rm log}_2 \hspace{0.1cm} (1/2) \hspace{0.15cm}\underline{= - 1\,{\rm bit}}\hspace{0.05cm}.$$

Javier: Text replacement - "value-discrete" to "discrete"

2021-10-11T08:27:33Z

Text replacement - "value-discrete" to "discrete"

Javier: Text replacement - "value-continuous" to "continuous"

2021-10-11T08:27:23Z

Text replacement - "value-continuous" to "continuous"

← Older revision		Revision as of 10:27, 11 October 2021
Line 14:		Line 14:


	The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated ~~value-~~continuous random variables.		The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.

	From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the page  [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .		From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the page  [[Information_Theory/Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .

	In the section  [[Information_Theory/Differentielle_Entropie#~~Entropy_of_value-continuous_random_variables_after_quantization~~\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:		In the section  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:
	:$$H(Z_{X, \hspace{0.05cm}M}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X)\hspace{0.05cm}. $$		:$$H(Z_{X, \hspace{0.05cm}M}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X)\hspace{0.05cm}. $$

	*In the course of the task it will be shown that in the case of rectangular PDF   ⇒   uniform distribution this  "approximation"  gives the same result as the direct calculation.		*In the course of the task it will be shown that in the case of rectangular PDF   ⇒   uniform distribution this  "approximation"  gives the same result as the direct calculation.
	*But in the general case – so in  [[Information_Theory/Differentielle_Entropie#~~Entropy_of_value-continuous_random_variables_after_quantization~~\|$\text{Example 2}$]]  with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy  $H(Z_{X,\hspace{0.05cm}M})$  only in the limiting case   ${\it \Delta} \to 0$ .		*But in the general case – so in  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|$\text{Example 2}$]]  with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy  $H(Z_{X,\hspace{0.05cm}M})$  only in the limiting case   ${\it \Delta} \to 0$ .


Line 33:		Line 33:
	Hints:		Hints:
	*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].		*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].
	*Useful hints for solving this task can be found in particular on the page  [[Information_Theory/Differentielle_Entropie#~~Entropy_of_value-continuous_random_variables_after_quantization~~\|Entropy of ~~value-~~continuous random variables after quantization]].		*Useful hints for solving this task can be found in particular on the page  [[Information_Theory/Differentielle_Entropie#Entropy_of_continuous_random_variables_after_quantization\|Entropy of continuous random variables after quantization]].

Javier: Text replacement - "continuous-valued" to "continuous"

2021-10-11T08:19:47Z

Text replacement - "continuous-valued" to "continuous"

← Older revision		Revision as of 10:19, 11 October 2021
Line 4:		Line 4:

	[[File:P_ID2878__Inf_A_4_4.png\|right\|frame\|Two uniform distributions]]		[[File:P_ID2878__Inf_A_4_4.png\|right\|frame\|Two uniform distributions]]
	We consider the two continuous~~-valued~~ random variables  $X$  and  $Y$  with probability density functions $f_X(x)$  and $f_Y(y)$.  For these random variables one can		We consider the two continuous random variables  $X$  and  $Y$  with probability density functions $f_X(x)$  and $f_Y(y)$.  For these random variables one can
	* not specify the conventional entropies  $H(X)$  and  $H(Y)$ , respectively,		* not specify the conventional entropies  $H(X)$  and  $H(Y)$ , respectively,
	* but the differential entropies  $h(X)$  and  $h(Y)$.		* but the differential entropies  $h(X)$  and  $h(Y)$.

Guenter at 14:45, 1 October 2021

2021-10-01T14:45:55Z

Guenter at 14:38, 1 October 2021

2021-10-01T14:38:47Z

← Older revision		Revision as of 16:38, 1 October 2021
Line 49:		Line 49:
	$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$		$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$

	{Calculate the entropy of the discrete ~~value~~ random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the direct method</u>.		{Calculate the entropy of the value-discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the direct method</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$		$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$

	{Calculate the entropy of the discrete ~~value~~ random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the given approximation</u>.		{Calculate the entropy of the value-discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the given approximation</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$		$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$

	{Calculate the entropy of the discrete ~~value~~ random variable  $Z_{Y,\hspace{0.05cm}M=8}$  with the given approximation</u>.		{Calculate the entropy of the value-discrete random variable  $Z_{Y,\hspace{0.05cm}M=8}$  with the given approximation</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{Y,\hspace{0.05cm}M=8})\ = \ $ { 3 3% } $\ \rm bit$		$H(Z_{Y,\hspace{0.05cm}M=8})\ = \ $ { 3 3% } $\ \rm bit$

	{Which of the following statements is true?		{Which of the following statements are true?
	\|type="[]"}		\|type="[]"}
	+ The entropy of a discrete value random variable  $Z$  is always  $H(Z) \ge 0$.		+ The entropy of a discrete value random variable  $Z$  is always  $H(Z) \ge 0$.

Noah at 10:54, 1 October 2021

2021-10-01T10:54:44Z

Show changes

← Older revision		Revision as of 17:57, 16 March 2026
Line 16:		Line 16:
	The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.		The probability density functions  $\rm (PDF)$  of these discrete random variables are each composed of  $M$  Dirac functions whose momentum weights are given by the interval areas of the associated continuous random variables.

	From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the section  [[Information_Theory/~~Einige_Vorbemerkungen_zu_zweidimensionalen_Zufallsgrößen~~#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .		From this, the entropies  $H(Z_{X,\hspace{0.05cm}M})$  and  $H(Z_{Y,\hspace{0.05cm}M})$  can be determined in the conventional way according to the section  [[Information_Theory/Some_Preliminary_Remarks_on_Two-Dimensional_Random_Variables#Probability_mass_function_and_entropy\|Probability mass function and entropy]] .

	In the section  [[Information_Theory/~~Differentielle_Entropie~~#Entropy_of_continuous_random_variables_after_quantization\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:		In the section  [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization\|Entropy of value-ontinuous random variables after quantization]],  an approximation was also given.  For example:
	:$$H(Z_{X, \hspace{0.05cm}M}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X)\hspace{0.05cm}. $$		:$$H(Z_{X, \hspace{0.05cm}M}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X)\hspace{0.05cm}. $$

	*In the course of the task it will be shown that in the case of rectangular PDF   ⇒   uniform distribution this  "approximation"  gives the same result as the direct calculation.		*In the course of the task it will be shown that in the case of rectangular PDF   ⇒   uniform distribution this  "approximation"  gives the same result as the direct calculation.
	*But in the general case – so in  [[Information_Theory/~~Differentielle_Entropie~~#Entropy_of_continuous_random_variables_after_quantization\|$\text{Example 2}$]]  with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy  $H(Z_{X,\hspace{0.05cm}M})$  only in the limiting case   ${\it \Delta} \to 0$ .		*But in the general case – so in  [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization\|$\text{Example 2}$]]  with triangular PDF – this equation is in fact only an approximation, which agrees with the actual entropy  $H(Z_{X,\hspace{0.05cm}M})$  only in the limiting case   ${\it \Delta} \to 0$ .


Line 33:		Line 33:
	Hints:		Hints:
	*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].		*The exercise belongs to the chapter  [[Information_Theory/Differentielle_Entropie\|Differential Entropy]].
	*Useful hints for solving this task can be found in particular in the section  [[Information_Theory/~~Differentielle_Entropie~~#Entropy_of_continuous_random_variables_after_quantization\|Entropy of continuous random variables after quantization]].		*Useful hints for solving this task can be found in particular in the section  [[Information_Theory/Differential_Entropy#Entropy_of_continuous_random_variables_after_quantization\|Entropy of continuous random variables after quantization]].


Line 124:		Line 124:

	[[Category:Information Theory: Exercises\|^4.1 Differential Entropy^]]		[[Category:Information Theory: Exercises\|^4.1 Differential Entropy^]]
	[[de:Aufgaben:4.~~04_Zusammenhang_h(X)_und_H(Z)~~]]		[[de:Aufgaben:Aufgabe 4.4: Herkömmliche Entropie und differenzielle Entropie]]

← Older revision		Revision as of 15:48, 16 March 2026
Line 89:		Line 89:

	Using the probability mass function, the <u>direct entropy calculation</u> gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:		Using the probability mass function, the <u>direct entropy calculation</u> gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:
	:$$H(Z_{X, \ M = 4}) = {\rm log}_2 \hspace{0.1cm} (4) \hspace{0.15cm}\underline{= 2\,{\rm bit}}		:$$H(Z_{X, \ M = 4}) = {\rm log}_2 \hspace{0.1cm} (4) \hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$

	With the <u>approximation</u>, considering the result of  '''(1)''', we obtain:		With the <u>approximation</u>, considering the result of  '''(1)''', we obtain:
	:$$H(Z_{X,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X) =		:$$H(Z_{X,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(X) =3\,{\rm bit} +(- 1\,{\rm bit})\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}. $$
	3\,{\rm bit} +(- 1\,{\rm bit})\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}. $$
	<i>Note:</i>  Only in the case of uniform distribution, the approximation gives exactly the same result as the direct calculation, i.e. the actual entropy.		<i>Note:</i>  Only in the case of uniform distribution, the approximation gives exactly the same result as the direct calculation, i.e. the actual entropy.

Line 103:		Line 101:
	* The possible values are now  $z \in \{\pm 0.75,\ \pm 0.25\}$.		* The possible values are now  $z \in \{\pm 0.75,\ \pm 0.25\}$.
	* Thus, here the  "approximation"  (as well as the direct calculation)  gives the result:		* Thus, here the  "approximation"  (as well as the direct calculation)  gives the result:
	:$$H(Z_{Y,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =		:$$H(Z_{Y,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =1\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}.$$
	1\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}.$$


	[[File:P_ID2881__Inf_A_4_4e.png\|right\|frame\|Quantized random variable  $Z_{Y, \ M = 8}$]]		[[File:P_ID2881__Inf_A_4_4e.png\|right\|frame\|Quantized random variable  $Z_{Y, \ M = 8}$]]
	'''(5)'''  In contrast to subtask  '''(5)''':   ${\it \Delta} = 1/4$  is now valid.  From this follows for the "approximation":		'''(5)'''  In contrast to subtask  '''(5)''':   ${\it \Delta} = 1/4$  is now valid.  From this follows for the "approximation":
	:$$H(Z_{Y,\hspace{0.05cm} M = 8}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =		:$$H(Z_{Y,\hspace{0.05cm} M = 8}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =2\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 3\,{\rm bit}}\hspace{0.05cm}.$$
	2\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 3\,{\rm bit}}\hspace{0.05cm}.$$
	Again, one gets the same result as in the direct calculation.		Again, one gets the same result as in the direct calculation.

← Older revision		Revision as of 10:27, 11 October 2021
Line 9:		Line 9:


	We also consider two ~~value-~~discrete random variables:		We also consider two discrete random variables:
	*The variable  $Z_{X,\hspace{0.05cm}M}$  is obtained by (suitably) quantizing the random quantity  $X$ with the quantization level number  $M$ <br>⇒   quantization interval width  ${\it \Delta} = 0.5/M$.		*The variable  $Z_{X,\hspace{0.05cm}M}$  is obtained by (suitably) quantizing the random quantity  $X$ with the quantization level number  $M$ <br>⇒   quantization interval width  ${\it \Delta} = 0.5/M$.
	* The variable  $Z_{Y,\hspace{0.05cm}M}$  is obtained after quantization of the random quantity  $Y$  with the quantization level number  $M$   <br>⇒   quantization interval width  ${\it \Delta} = 2/M$.		* The variable  $Z_{Y,\hspace{0.05cm}M}$  is obtained after quantization of the random quantity  $Y$  with the quantization level number  $M$   <br>⇒   quantization interval width  ${\it \Delta} = 2/M$.
Line 49:		Line 49:
	$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$		$ h(Y) \ = \ $ { 1 3% } $\ \rm bit$

	{Calculate the entropy of the ~~value-~~discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the direct method</u>.		{Calculate the entropy of the discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the direct method</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$		$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$

	{Calculate the entropy of the ~~value-~~discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the given approximation</u>.		{Calculate the entropy of the discrete random variables  $Z_{X,\hspace{0.05cm}M=4}$  using the given approximation</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$		$H(Z_{X,\hspace{0.05cm}M=4})\ = \ $ { 2 3% } $\ \rm bit$

	{Calculate the entropy of the ~~value-~~discrete random variable  $Z_{Y,\hspace{0.05cm}M=8}$  with the given approximation</u>.		{Calculate the entropy of the discrete random variable  $Z_{Y,\hspace{0.05cm}M=8}$  with the given approximation</u>.
	\|type="{}"}		\|type="{}"}
	$H(Z_{Y,\hspace{0.05cm}M=8})\ = \ $ { 3 3% } $\ \rm bit$		$H(Z_{Y,\hspace{0.05cm}M=8})\ = \ $ { 3 3% } $\ \rm bit$
Line 115:		Line 115:

	'''(6)'''  Only <u>statement 1</u>  is correct:		'''(6)'''  Only <u>statement 1</u>  is correct:
	* The entropy  $H(Z)$  of a ~~value-~~discrete random variable  $Z = \{z_1, \ \text{...} \ , z_M\}$  is never negative.		* The entropy  $H(Z)$  of a discrete random variable  $Z = \{z_1, \ \text{...} \ , z_M\}$  is never negative.
	*For example, the limiting case  $H(Z) = 0$  results for  ${\rm Pr}(Z = z_1) = 1$  and  ${\rm Pr}(Z = z_\mu) = 0$  for  $2 \le \mu \le M$.		*For example, the limiting case  $H(Z) = 0$  results for  ${\rm Pr}(Z = z_1) = 1$  and  ${\rm Pr}(Z = z_\mu) = 0$  for  $2 \le \mu \le M$.

← Older revision		Revision as of 16:45, 1 October 2021
Line 85:		Line 85:
	'''(3)'''  The adjacent graph illustrates the best possible quantization of random variable  $X$  with quantization level number  $M = 4$    ⇒   random variable  $Z_{X, \ M = 4}$:		'''(3)'''  The adjacent graph illustrates the best possible quantization of random variable  $X$  with quantization level number  $M = 4$    ⇒   random variable  $Z_{X, \ M = 4}$:
	*The interval width here is equal to  ${\it \Delta} = 0.5/4 = 1/8$.		*The interval width here is equal to  ${\it \Delta} = 0.5/4 = 1/8$.
	*The possible values  (at the center of the interval, respectively)  are  $z \in \{0.0625,\ 0.1875,\ 0.3125,\ 0.4375\}$.		*The possible values  (at the center of the interval,  respectively)  are  $z \in \{0.0625,\ 0.1875,\ 0.3125,\ 0.4375\}$.


	Using the probability function, the <u>direct entropy calculation</u> gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:		Using the probability mass function, the <u>direct entropy calculation</u> gives $P_Z(Z) = \big [1/4,\ \text{...} , \ 1/4 \big]$:
	:$$H(Z_{X, \ M = 4}) = {\rm log}_2 \hspace{0.1cm} (4) \hspace{0.15cm}\underline{= 2\,{\rm bit}}		:$$H(Z_{X, \ M = 4}) = {\rm log}_2 \hspace{0.1cm} (4) \hspace{0.15cm}\underline{= 2\,{\rm bit}}
	\hspace{0.05cm}.$$		\hspace{0.05cm}.$$
Line 102:		Line 102:
	* The quantization parameter is now  ${\it \Delta} = 2/4 = 1/2$.		* The quantization parameter is now  ${\it \Delta} = 2/4 = 1/2$.
	* The possible values are now  $z \in \{\pm 0.75,\ \pm 0.25\}$.		* The possible values are now  $z \in \{\pm 0.75,\ \pm 0.25\}$.
	* Thus, here the "approximation"  (as well as the direct calculation)  gives the result:		* Thus, here the  "approximation"  (as well as the direct calculation)  gives the result:
	:$$H(Z_{Y,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =		:$$H(Z_{Y,\hspace{0.05cm} M = 4}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =
	1\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}.$$		1\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 2\,{\rm bit}}\hspace{0.05cm}.$$
Line 108:		Line 108:

	[[File:P_ID2881__Inf_A_4_4e.png\|right\|frame\|Quantized random variable  $Z_{Y, \ M = 8}$]]		[[File:P_ID2881__Inf_A_4_4e.png\|right\|frame\|Quantized random variable  $Z_{Y, \ M = 8}$]]
	'''(5)'''  In contrast to subtask  '''(5)'''   ${\it \Delta} = 1/4$ is now valid.  From this follows for the "approximation":		'''(5)'''  In contrast to subtask  '''(5)''':   ${\it \Delta} = 1/4$  is now valid.  From this follows for the "approximation":
	:$$H(Z_{Y,\hspace{0.05cm} M = 8}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =		:$$H(Z_{Y,\hspace{0.05cm} M = 8}) \approx -{\rm log}_2 \hspace{0.1cm} ({\it \Delta}) + h(Y) =
	2\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 3\,{\rm bit}}\hspace{0.05cm}.$$		2\,{\rm bit} + 1\,{\rm bit}\hspace{0.15cm}\underline{= 3\,{\rm bit}}\hspace{0.05cm}.$$
	Again, one gets the same result as in the direct calculation.		Again, one gets the same result as in the direct calculation.
	~~<br clear=all>~~
	'''(6)'''  Only <u>statement 1</u> is correct:
	* The entropy  $H(Z)$  of a discrete random variable  $Z = \{z_1, \ \text{...} \ , z_M\}$  is never negative.		'''(6)'''  Only <u>statement 1</u>  is correct:
			* The entropy  $H(Z)$  of a value-discrete random variable  $Z = \{z_1, \ \text{...} \ , z_M\}$  is never negative.
	*For example, the limiting case  $H(Z) = 0$  results for  ${\rm Pr}(Z = z_1) = 1$  and  ${\rm Pr}(Z = z_\mu) = 0$  for  $2 \le \mu \le M$.		*For example, the limiting case  $H(Z) = 0$  results for  ${\rm Pr}(Z = z_1) = 1$  and  ${\rm Pr}(Z = z_\mu) = 0$  for  $2 \le \mu \le M$.