Section 7.5 Generating Functions for Bernoulli-based Distributions
Theorem 7.5.1. Moment Generating Function for Bernoulli.
\begin{equation*}
M(t) = (1-p) + p e^t
\end{equation*}
Proof.
\begin{equation*}
M(t) = f(0) + e^t f(1) = (1-p) + p e^t
\end{equation*}
Theorem 7.5.2. Moment Generating Function for Geometric.
\begin{equation*}
M(t) = \frac{p e^t }{1 - e^t (1-p)} = \frac{p}{e^{-t} - (1-p)}
\end{equation*}
Proof.
Presuming \(e^t (1-p) \lt 1\text{,}\)
\begin{align*}
M(t) & = \sum_{x=1}^{\infty} e^{tx} p (1-p)^{x-1}\\
& = \frac{p}{1-p} \sum_{x=1}^{\infty} (e^t (1-p))^x\\
& = \frac{p}{1-p} \frac{e^t (1-p)}{1 - e^t (1-p)}\\
& = \frac{pe^t }{1 - e^t (1-p)}.
\end{align*}
where we used the geometric series to convert the sum. The second form comes by dividing through by \(e^t\text{.}\)
Corollary 7.5.3. Geometric Properties via Moment Generating Function.
For the Geometric variable X,
\begin{equation*}
M(0) = 1
\end{equation*}
\begin{equation*}
M'(0) = \frac{1}{p} = \mu
\end{equation*}
\begin{equation*}
M''(0) = \frac{1-p}{p^2} + \frac{1}{p^2} = \sigma^2 + \mu^2
\end{equation*}
Proof.
\begin{equation*}
M(0) = p \frac{e^0 }{1 - e^0 (1-p)} = \frac{p}{1-(1-p)} = 1.
\end{equation*}
Using the second form for M(t),
\begin{equation*}
M'(t) = \frac{e^{-t} p}{(e^{-t} - (1-p))^2}
\end{equation*}
and therefore
\begin{equation*}
M'(0) = \frac{p}{(1-(1-p))^2} = \frac{1}{p}.
\end{equation*}
Continuing with the second derivative,
\begin{equation*}
M''(t) = -\frac{p e^{-t} }{{\left(p + e^{-t} - 1\right)}^2} + \frac{2 p e^{-2t}}{{\left(p + e^{-t} - 1 \right)}^{3}}
\end{equation*}
and therefore
\begin{equation*}
M''(0) = -\frac{p}{{\left(p + 1 - 1 \right)}^2} + \frac{2 p}{{\left(p + 1 - 1 \right)}^{3}} = -\frac{1}{p} + \frac{2}{p^2} = \frac{1}{p^2} + \frac{1-p}{p^2}
\end{equation*}
which is the squared mean plus the variance for the geometric distribution.
For the two uniform distributions and basic Bernoulli, there is really nothing much that can be done as n (or a and b) vary. However, for all distributions that have the opportunity to "scale" larger and larger, we will first determine the M(t) and then demonstrate the a surprising relationship as their variables to grow appropriately.
Theorem 7.5.4. Moment Generating Function for Binomial.
\begin{equation*}
M(t) = \left ( p e^t + (1-p) \right )^n
\end{equation*}
Proof.
\begin{align*}
M(t) & = \sum_{x=0}^n e^{tx} \binom{n}{x} p^x (1-p)^{n-x} \\
& = \sum_{x=0}^n \binom{n}{x} (pe^t)^x (1-p)^{n-x} \\
& = \left ( p e^t + (1-p) \right )^n
\end{align*}
where we used the binomial theorem to simplify the sum.
Notice that the moment generating function for Bernoulli is simply the Binomial moment generating function with n=1.
Corollary 7.5.5. Binomial Properties via Moment Generating Function.
\begin{equation*}
M(0) = 1
\end{equation*}
\begin{equation*}
M'(0) = np
\end{equation*}
\begin{equation*}
M''(0) = np(1-p) + (np)^2
\end{equation*}
Proof.
\begin{equation*}
M(0) = \left ( p e^0 + (1-p) \right )^n = 1^n = 1.
\end{equation*}
Taking the derivative with respect to t,
\begin{equation*}
M'(t) = n \left ( p e^t + (1-p) \right )^{n-1} p e^t
\end{equation*}
and evaluating at t=0 gives
\begin{equation*}
M'(0) = n \left ( p + (1-p) \right )^{n-1} p = n 1^{n-1} p = np.
\end{equation*}
Again, taking another derivative with respect to t,
\begin{equation*}
M''(t) = n(n-1) \left ( p e^t + (1-p) \right )^{n-2} p^2 e^{2t} + n \left ( p e^t + (1-p) \right )^{n-1} p e^t
\end{equation*}
and evaluating at t=0 gives
\begin{align*}
M''(0) & = n(n-1) ( p + (1-p))^{n-2} p^2 + n ( p + (1-p) )^{n-1} p \\
& = n(n-1)p^2 + np = (np)^2 + np - np^2 = np(1-p) + (np)^2.
\end{align*}
Theorem 7.5.6. Moment Generating Function for Negative Binomial.
\begin{equation*}
M(t) = \frac{p^r}{(1 - e^t(1-p))^r}
\end{equation*}
Proof.
Using the
a previous theorem 7.4.5 justifying the Negative Binomial probability function with
\(a = p e^t\) and
\(b = 1-p\) and by changing variables to
\(u = x-r\) gives
\begin{align*}
M(t) & = \sum_{u=0}^{\infty} e^{tu} \binom{u+r - 1}{r-1}(1-p)^{u}p^r\\
& = p^r \sum_{u=0}^{\infty} \binom{u + r - 1}{r-1}(e^t(1-p))^u \\
& = \frac{ p^{r}}{(1 - e^t(1-p))^r} \sum_{u=0}^{\infty} \binom{u + r - 1}{r-1}(1 - e^t(1-p))^r (e^t(1-p))^u \\
& = \frac{p^{r}}{(1 - e^t(1-p))^r}
\end{align*}
noting that the last summation is the the sum of a negative binomial probability function over its entire range.
Notice that the moment generating function for Geometric is simply the Negative Binomial moment generating function with r=1.
Corollary 7.5.7. Negative Binomial Properties via Moment Generating Function.
\begin{equation*}
M(0) = 1
\end{equation*}
\begin{equation*}
M'(0) = \frac{r}{p}
\end{equation*}
\begin{equation*}
M''(0) = \left ( \frac{r(1-p)}{p^2} \right )^2 + \left ( \frac{r}{p} \right )^2
\end{equation*}
Proof.
\begin{equation*}
M(0) = \frac{(p)^r}{(1 - (1-p))^r} = \frac{p^r}{p^r} = 1.
\end{equation*}
Taking the derivative with respect to t,
\begin{equation*}
M'(t) = -\frac{{\left(p e^{t} - e^{t} + 1\right)}^{r - 1} {\left(p - 1\right)} p^{r} r e^{t}}{{\left(p e^{t} - e^{t} + 1\right)}^{2 r}}
\end{equation*}
and evaluating at t=0 gives
\begin{equation*}
M'(0) = -\frac{{p}^{r - 1} {p - 1} p^{r} r }{{p}^{2 r}} = \frac{r(1-p)}{p}.
\end{equation*}
OOPS...don’t need the (1-p) on top. Again, taking another derivative with respect to t,
\begin{equation*}
M''(t) = -\frac{{\left(p e^{t} - e^{t}\right)} {\left(p e^{t} - e^{t} + 1\right)}^{r - 2} {\left(p - 1\right)} p^{r} {\left(r - 1\right)} r e^{t}}{{\left(p e^{t} - e^{t} + 1\right)}^{2r}}
\\ + \frac{2 {\left(p e^{t} - e^{t}\right)} {\left(p e^{t} - e^{t} + 1\right)}^{2r - 2} {\left(p - 1\right)} p^{r} r^{2} e^{t}}{{\left(p e^{t} - e^{t} + 1\right)}^{3r}}
\\ - \frac{{\left(p e^{t} - e^{t} + 1\right)}^{r - 1} {\left(p - 1\right)} p^{r} r e^{t}}{{\left(p e^{t} - e^{t} + 1\right)}^{2r}}
\end{equation*}
and evaluating at t=0 gives
\begin{align*}
M''(0) & = \frac{{\left(p^{2 \, r - 1} r - p^{2 \, r - 2} r - p^{2 \, r - 2}\right)} {\left(p - 1\right)} r}{p^{2 \, r}} \\
& = \left ( \frac{r(1-p)}{p^2} \right )^2 + \left ( \frac{r}{p} \right )^2.
\end{align*}
Just in case you are wondering about the derivations and especially taking the derivatives above, Sage will do the heavy lifting for you.