Probability Functions

Section 5.3 Probability Functions

In the formulas below, we will presume that we have a random variable X which maps the sample space S onto some range of real numbers R. From this set, we then can define a probability function f(x) which acts on the numerical values in R and returns another real number. We attempt to do so to obtain (for discrete values) P(sample space value s) $= f(X(s))\text{.}$ That is, the probability of a given outcome s is equal to the composition which takes s to a numerical value x which is then plugged into f to get the same final values.

For example, consider a random variable which assigns a 1 when you roll a 1 on a six-sided die and 0 otherwise. Presuming each side is equally likely,

Definition 5.3.1 Probability "Mass" Function

Given a discrete random variable X on a space R, a probability mass function on X is given by a function

$f:R \rightarrow \mathbb{R}$ such that:

$\begin{align*} & \forall x \in R , f(x) \gt 0\\ & \sum_{x \in R} f(x) = 1\\ & A \subset R \Rightarrow P(X \in A) = \sum_{x \in A}f(x) \end{align*}$

For

$x \not\in R\text{,}$ you can use the convention f(x)=0.

Definition 5.3.2 Probability "Density" Function

Given a continuous random variable X on a space R, a probability density function on X is given by a function

$f:R \rightarrow \mathbb{R}$ such that:

$\begin{align*} & \forall x \in R , f(x) \gt 0\\ & \int_{R} f(x) dx = 1\\ & A \subset R \Rightarrow P(X \in A) = \int_{A} f(x) dx \end{align*}$

For

$x \not\in R\text{,}$ you can use the convention f(x)=0.

For the purposes of this book, we will use the term "Probability Function" to refer to either of these options.

Example 5.3.3 Discrete Probability Function

Consider $f(x) = x/10$ over R = {1,2,3,4}. Then, f(x) is obviously positive for each of the values in R and certainly $\sum_{x \in R} f(x) = f(1) + f(2) + f(3) + f(4) = 1/10 + 2/10 + 3/10 + 4/10 = 1\text{.}$ Therefore, f(x) is a probability mass function over the space R.

xxxxxxxxxx
 
# Combining all of the above into one interactive cell
@interact
def _(D = input_box([1,2,3,5,6,8,9,11,12,14],label="Enter domain R (in brackets):"), 
       Probs = input_box([1/20,1/20,1/20,3/20,1/20,4/20,4/20,1/20,1/20,3/20],label="Enter corresponding f(x) (in brackets):"),
       n_samples=slider(100,10000,100,100,label="Number of times to sample from this distribution:")):
    n = len(D)
    R = range(n)
    one_huh = sum(Probs)
    pretty_print('\n\nJust to be certain, we should check to make certain the probabilities sum to 1\n')
    pretty_print(html('$\sum_{x\epsilon R} f(x) = %s$'%str(one_huh)))
    
    G = Graphics()
    if len(D)==len(Probs):
        f = zip(D,Probs)
        meanf = 0
        variancef = 0
        for k in R:
            meanf += D[k]*Probs[k]
            variancef += D[k]^2*Probs[k]
            G += line([(D[k],0),(D[k],Probs[k])],color='green')
        variancef = variancef - meanf^2
        sd = sqrt(variancef)
        G += points(f,color='blue',size=50)
        G += point((meanf,0),color='yellow',size=60,zorder=3)
        G += line([(meanf-sd,0),(meanf+sd,0)],color='red',thickness=5)
    
        g = DiscreteProbabilitySpace(D,Probs)
        pretty_print('     mean = %s'%str(meanf))
        pretty_print(' variance = %s'%str(variancef))
    
        #  perhaps to add mean and variance for pmf here
    else:
        print 'Domain D and Probabilities Probs must be lists of the same size'
    
    #  Now, let's sample from the distribution given above and see how a random sampling matches up
​
    counts = [0] * len(Probs)
    X = GeneralDiscreteDistribution(Probs)
    sample = []
​
    for _ in range(n_samples):
        elem = X.get_random_element()
        sample.append(D[elem])
        counts[elem] += 1
    Empirical = [1.0*x/n_samples for x in counts] # random
    
    samplemean = mean(sample)
    samplevariance = variance(sample)
    sampdev = sqrt(samplevariance)
    
    E = points(zip(D,Empirical),color='orange',size=40)
    E += point((samplemean,0.005),color='brown',size=60,zorder=3)
    E += line([(samplemean-sampdev,0.005),(samplemean+sampdev,0.005)],color='orange',thickness=5)    
    (G+E).show(ymin=0,figsize=(8,5))

Example 5.3.4 Continuous Probability Function

Consider $f(x) = x^2/c$ for some positive real number c and presume R = [-1,2]. Then f(x) is nonnegative (and only equals zero at one point). To make f(x) a probability density function, we must have

\begin{equation*} \int_{x \in R} f(x) = 1. \end{equation*}

In this instance you get

\begin{equation*} 1 = \int_{-1}^2 x^2/c = x^3/(3c) |_{-1}^2 = \frac{8}{3c} - \frac{-1}{3c} = \frac{3}{c} \end{equation*}

Therefore, f(x) is a probability density function over R provided = 3.

Definition 5.3.5 Distribution Function

Given a random variable X on a space R, a probability distribution function on X is given by a function

$F:\mathbb{R} \rightarrow \mathbb{R}$ such that

$\displaystyle F(x)=P(X \le x)$

Example 5.3.6 Discrete Distribution Function

Using $f(x) = x/10$ over R = {1,2,3,4} again, note that F(x) will only change at these four domain values. We get

X	F(x)
$x \lt 1$	0
$1 \le x \lt 2$	1/10
$2 \le x \lt 3$	3/10
$3 \le x \lt 4$	6/10
$4 \le x $	1

Table 5.3.7 Discrete Distribution Function Example

Example 5.3.8 Continuous Distribution Function

Consider $f(x) = x^2/3$ over R = [-1,2]. Then, for $-1 \le x \le 2\text{,}$

\begin{equation*} F(x) = \int_{-1}^x u^2/3 du = x^3/9 + 1/9. \end{equation*}

Notice, F(-1) = 0 since nothing has yet been accumulated over values smaller than -1 and F(2)=1 since by that time everything has been accumulated. In summary:

X	F(x)
$x \lt -1$	0
$-1 \le x \lt 2$	$x^3/9 + 1/9$
$2 \le x$	1

Table 5.3.9 Continuous Distribution Function Example

Theorem 5.3.10

$F(x)=0, \forall x \lt \inf(R)$

Proof

Let a = inf(R). Then, for $x \lt a,$

\begin{equation*} F(x) = P(X \le x) \le P(X \lt a) = 0 \end{equation*}

since none of the x-values in this range are in R.

Theorem 5.3.11

$F(x)=1, \forall x \ge \sup(R)$

Proof

Let b = sup(R). Then, for

\begin{equation*} x \ge b, F(x) = P(X \le x) = P(X \le b) + P( b \lt X \le x) = P(X \le b) = 1 \end{equation*}

since all of the x-values in this range are in R and therefore will either sum over or integrate over all of R.

Theorem 5.3.12

F is non-decreasing

Proof

Case 1: R discrete

\begin{align*} \forall x_1,x_2 \in \mathbb{Z} \ni x_1 \lt x_2\\ F(x_2) & = \sum_{x \le x_2} f(x) \\ & = \sum_{x \le x_1} f(x) + \sum_{x_1 \lt x \le x_2} f(x)\\ & \ge \sum_{x \le x_1} f(x) = F(x_1) \end{align*}

Case 2: R continuous

\begin{align*} \forall x_1,x_2 \in \mathbb{R} \ni x_1 \lt x_2\\ F(x_2) & = \int_{-\infty}^{x_2} f(x) dx \\ & = \int_{-\infty}^{x_1} f(x) dx + \int_{x_1}^{x_2} f(x) dx\\ & \ge \int_{-\infty}^{x_1} f(x) dx\\ & = F(x_1) \end{align*}

Theorem 5.3.13 Using Discrete Distribution Function to compute probabilities

for

$x \in R, f(x) = F(x) - F(x-1)$

Proof

Assume $x \in R$ for some discrete R. Then,

\begin{equation*} F(x) - F(x-1) = \sum_{u \le x} f(u) - \sum_{u \lt x} f(u) = f(x) \end{equation*}

Theorem 5.3.14 Using Continuous Distribution function to compute probabilities

for

$a \lt b, (a,b) \in R, P(a \lt X \le b) = F(b) - F(a)$

Proof

For a and b as noted, consider

\begin{align*} F(b) - F(a) & = \int_{-\infty}^b f(x) dx - \int_{-\infty}^a f(x) dx\\ & = \int_a^b f(x) dx \\ & = P(a \lt x \le b) \end{align*}

Corollary 5.3.15

For continuous distributions, P(X = a) = 0

Proof

We will assume that F(x) is a continuous function. With that assumption, note

\begin{equation*} P(a-\epsilon \lt x \le a) = \int_{a-\epsilon}^a f(x) dx = F(a) - F(a-\epsilon) \end{equation*}

Take the limit as $\epsilon \rightarrow 0^+$ to get the result noting that

Theorem 5.3.16 F(x) vs f(x), for continuous distributions

If X is a continuous random variable, f the corresponding probability function, and F the associated distribution function, then

$\begin{equation*} f(x) = F'(x) \end{equation*}$

Proof

Assume X is continuous and f and F as above. Notice, by the definition of f, $\lim_{x \rightarrow \pm \infty} f(x) = 0$ since otherwise the integral over the entire space could not be finite.

Now, let A(x) be any antiderivative of f(x). Then, by the Fundamental Theorem of Calculus,

\begin{align*} F(x) & = \int_{-\infty}^x f(u) du\\ & = A(x) - \lim_{u \rightarrow -\infty} A(u) \end{align*}

Hence, $F'(x) = A'(x) - \lim_{u \rightarrow -\infty} A'(u) = f(x)$ as desired.

X	F(x)
\(x \lt 1\)	0
\(1 \le x \lt 2\)	1/10
\(2 \le x \lt 3\)	3/10
\(3 \le x \lt 4\)	6/10
\(4 \le x \)	1

X	F(x)
\(x \lt -1\)	0
\(-1 \le x \lt 2\)	\(x^3/9 + 1/9\)
\(2 \le x\)	1