Expectation over a Gaussion Distribution

Published on: Date

Following our weekly book reading session at NAAMII, where we discussed Chapter 2: Probabilities from the book 'Deep Learning - Foundations and Concepts,' we decided to explore the concept of expectation in different probability distributions. This was a great chance to remember and brush up on our integration lessons!

Expected value of function of a continuous variable x is given by

E (f) = \int p (x) . f (x) d x

Which can be viewed as a weighted average of function

f (x)

under a probability distribution

p (x)

.

Gaussian distribution is defined by:

p (x) = \frac{1}{σ \sqrt{2 π}} * e^{- \frac{1}{2} * (\frac{x - μ}{σ})^{2}}

Here, probability density over x is governed by two parameters:

μ

, called the mean,

σ

, called the standarad deviation. The distribution is defined over entire real line, extending from

- \infty

\infty

let

f (x) = x

E (x) = \int_{- \infty}^{\infty} \frac{1}{σ \sqrt{2 π}} * e^{- \frac{1}{2} * (\frac{x - μ}{σ})^{2}} * x d x

----------(1)

Let:

\frac{x - μ}{σ} = a

and,

x = a σ + μ

d x = σ d a

Put the value of x in equation 1.

= \int_{- \infty}^{\infty} \frac{1}{σ * \sqrt{2 π}} * e^{- \frac{1}{2} a^{2}} * (a σ + μ) * σ * d a

= \frac{1}{\sqrt{2 π}} * [\int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * a * σ * d a + \int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * μ * d a]

------------ (2)

Let:

J = \int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * a * σ * d a

I = \int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * μ * d a

Solving for J:

= \int_{- \infty}^{0} e^{- \frac{1}{2} a^{2}} * a * σ * d a + \int_{0}^{\infty} e^{- \frac{1}{2} a^{2}} * a * σ * d a

= - \int_{0}^{\infty} e^{- \frac{1}{2} a^{2}} * a * σ * d a + \int_{0}^{\infty} e^{- \frac{1}{2} a^{2}} * a * σ * d a

= 0

Solving for I:

I = \int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * μ * d a

I^{2} = \int_{- \infty}^{\infty} e^{- \frac{1}{2} a^{2}} * μ * d a * \int_{- \infty}^{\infty} e^{- \frac{1}{2} b^{2}} * μ * d b

I^{2} = \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} e^{- \frac{1}{2} (a^{2} + b^{2})} * μ^{2} * d a d b

Next we change cartesian to polar form:

a^{2} + b^{2} = r^{2}

d a * d b = r * d r * d θ

Therefore:

= \int_{0}^{2 π} \int_{0}^{\infty} e^{- \frac{1}{2} (r^{2})} * μ^{2} * r * d r * d θ

= 2 π * \int_{0}^{\infty} e^{- \frac{1}{2} (r^{2})} * μ^{2} * r * d r

Let:

r^{2} = x

2 * r * d r = d x

Now,

= π * \int_{0}^{\infty} e^{- \frac{x}{2}} * μ^{2} * d x

= π * (\frac{e^{- \frac{\infty}{2}}}{- \frac{1}{2}} - \frac{e^{- \frac{0}{2}}}{- \frac{1}{2}}) * μ^{2}

= π * (0 + 2) * μ^{2}

So,

I^{2} = 2 π * μ^{2}

I = \sqrt{2 π * μ^{2}}

I = μ \sqrt{2 π}

Keeping the value of I and J back to equation 2 we get,

E (x) = \frac{1}{\sqrt{2 π}} * \sqrt{2 π} * μ

E (x) = μ