Chapter 3 - Random Variables and Their Distributions

Entry | Solutions

Random Variables

Definition

Random Variable

Given an experiment with sample space $S$, a random variable (r.v.) is a function from the sample space $S$ to the real numbers $R$. It is common, but not required, to denote random variables by capital letters. Thus, a random variable $X$ assigns a numerical value $X(s)$ to each possible outcome s of the experiment. The randomness comes from the fact that we have a random experiment (with probabilities described by the probability function $P$); the mapping itself is deterministic.

Example

Consider an experiment where we toss a fair coin twice. The sample space consists of four possible outcomes: $S=\{HH,HT,TH,TT\}$. Here are some random variables on this space (for practice, you can think up some of your own). Each r.v. is a numerical summary of some aspect of the experiment. Let $X$ be the number of Heads. This is a random variable with possible values $0$, $1$, and $2$. Viewed as a function, $X$ assigns the value $2$ to the outcome $HH$, $1$ to the outcomes $HT$ and $TH$, and $0$ to the outcome $TT$. That is,

$$X(HH)=2,X(HT)=X(TH)=1,X(TT)=0.$$

Distributions and Probability Mass Functions

Definition

Discrete Random Variable

A random variable $X$ is said to be discrete if there is a finite list of values $a_1,a_2,...,a_n$ or an infinite list of values $a_1,a_2,\dots$ such that $P(X=a_j\text{ for some j})=1$. If $X$ is a discrete r.v., then the finite or countably infinite set of values $x$ such that $P(X=x)>0$ is called the support of $X$.

Definition

Probability Mass Function

The probability mass function (PMF) of a discrete r.v. $X$ is the function $p_X$ given by $p_X(x)=P(X=x)$. Note that this is positive if $x$ is in the support of $X$, and $0$ otherwise.

Note

In writing $P(X=x)$, we are using $X=x$ to denote an event, consisting of all outcomes $s$ to which $X$ assigns the number $x$. This event is also written as $X=x$; formally, $X=x$ is defined as $s\in S:X(s)=x$, but writing $X=x$ is shorter and more intuitive.

Theorem

Valid PMFs

Let $X$ be a discrete r.v. with support $x_1,x_2,...$ (assume these values are distinct and, for notational simplicity, that the support is countably infinite; the analogous results hold if the support is finite). The PMF $p_X$ of $X$ must satisfy the following two criteria:

• Nonnegative: $p_X(x)>0$ if $x=x_j$ for some $j$, and $p_X(x)=0$ otherwise;
• Sums to $1$: ${\sum }_{j=1}^{\infty }{p}_X(x_j)=1$.

Bernoulli and Binomial

Definition

Bernoulli Distribution

An r.v. $X$ is said to have the Bernoulli distribution with parameter $p$ if $P(X=1)=p$ and $P(X=0)=1-p$, where $0<p<1$. We write this as $X\sim Bern(p)$. The symbol $\sim$ is read “is distributed as”.

Story

Bernoulli Trial An experiment that can result in either a “success” or a “failure” (but not both) is called a Bernoulli trial. A Bernoulli random variable can be thought of as the indicator of success in a Bernoulli trial: it equals $1$ if success occurs and $0$ if failure occurs in the trial.

Definition

Indicator Random Variable

The indicator random variable of an event $A$ is the r.v. which equals $1$ if $A$ occurs and $0$ otherwise. We will denote the indicator r.v. of $A$ by $I_A$ or $I(A)$. Note that $I_A\sim Bern(p)$ with $p=P(A)$.

Story

Binomial Distribution

Suppose that $n$ independent Bernoulli trials are performed, each with the same success probability $p$. Let $X$ be the number of successes. The distribution of $X$ is called the Binomial distribution with parameters $n$ and $p$. We write $X\sim Bin(n,p)$ to mean that $X$ has the Binomial distribution with parameters $n$ and $p$, where $n$ is a positive integer and $0<p<1$.

Note

it is clear that $Bern(p)$ is the same distribution as $Bin(1,p)$: the Bernoulli is a special case of the Binomial.

Theorem

Binomial PMF

If $X\sim Bin(n,p)$, then the PMF of $X$ is

$$P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$$

for $k=0,1,...,n$ (and $P(X=k)=0$ otherwise).

Theorem

Let $X\sim Bin(n,p)$, and $q=1-p$ (we often use $q$ to denote the failure probability of a Bernoulli trial). Then $n-X\sim Bin(n,q)$.

Corollary

Let $X\sim Bin(n,p)$ with $p=1/2$ and $n$ even. Then the distribution of $X$ is symmetric about $n/2$, in the sense that $P(X=n/2+j)=P(X=n/2-j)$ for all nonnegative integers $j$.

Hypergeometric

Story

Hypergeometric Distribution

Consider an urn with $w$ white balls and $b$ black balls. We draw $n$ balls out of the urn at random without replacement, such that all $\left(\genfrac{}{}{0ex}{}{w+b}{n}\right)$ samples are equally likely. Let $X$ be the number of white balls in the sample. Then $X$ is said to have the Hypergeometric distribution with parameters $w$, $b$, and $n$; we denote this by $X\sim HGeom(w,b,n)$.

Theorem

Hypergeometric PMF

If $X\sim HGeom(w,b,n)$, then the PMF of $X$ is

$$P(X=k)=\frac{\left(\genfrac{}{}{0ex}{}{w}{k}\right)\left(\genfrac{}{}{0ex}{}{b}{n-k}\right)}{\left(\genfrac{}{}{0ex}{}{w+b}{n}\right)}$$

for integers $k$ satisfying $0\le k\le w$ and $0\le n-k\le b$, and $P(X=k)=0$ otherwise.

Theorem

The $HGeom(w,b,n)$ and $HGeom(n,w+b-n,w)$ distributions are identical. That is, if $X\sim HGeom(w,b,n)$ and $Y\sim HGeom(n,w+b-n,w)$, then $X$ and $Y$ have the same distribution.

Discrete Uniform

Story

Discrete Uniform Distribution

Let $C$ be a finite, nonempty set of numbers. Choose one of these numbers uniformly at random (i.e., all values in $C$ are equally likely). Call the chosen number $X$. Then $X$ is said to have the Discrete Uniform distribution with parameter $C$; we denote this by $X\sim DUnif(C)$.

Theorem

The PMF of $X\sim DUnif(C)$ is

$$P(X=x)=\frac{1}{|C|}$$

for $x\in C$ (and $0$ otherwise), since a PMF must sum to $1$. As with questions based on the naive definition of probability, questions based on a Discrete Uniform distribution reduce to counting problems. Specifically, for $X\sim DUnif(C)$ and any $A\subseteq C$, we have

$$P(X\in A)=\frac{|A|}{|C|}$$

Cumulative Distribution Functions

Definition

The cumulative distribution function (CDF) of an r.v. $X$ is the function $F_X$ given by $F_X(x)=P(X\le x)$. When there is no risk of ambiguity, we sometimes drop the subscript and just write $F$ (or some other letter) for a CDF.

Theorem

Valid CDFs

Any CDF $F$ has the following properties.

• Increasing: If $x_1\le x_2$, then $F(x_1)\le F(x_2)$.
• Right-continuous: The CDF is continuous except possibly for having some jumps. Wherever there is a jump, the CDF is continuous from the right. That is, for any $a$, we have

$$F(a)=\underset{x\to a^{+}}{\lim }F(x)$$

• Convergence to $0$ and $1$ in the limits:

$$\underset{x\to -\infty }{\lim }F(x)=0\text{ and }\underset{x\to \infty }{\lim }F(x)=1$$

Functions of Random Variables

Definition

Function of an r.v.

For an experiment with sample space $S$, an r.v. $X$, and a function $g:R\to R$, $g(X)$ is the r.v. that maps s to $g(X(s))$ for all $s\in S$.

Theorem

PMF of $g(X)$

Let X be a discrete r.v. and $g:R\to R$. Then the support of $g(X)$ is the set of all $y$ such that $g(x)=y$ for at least one $x$ in the support of $X$, and the PMF of $g(X)$ is

$$P(g(X)=y)=\sum _{x:g(x)=y}P(X=x)$$

for all $y$ in the support of $g(X)$

Definition

Functions of two r.v.s

Given an experiment with sample space $S$, if $X$ and $Y$ are r.v.s that map $s\in S$ to $X(s)$ and $Y(s)$ respectively, then $g(X,Y)$ is the r.v. that maps $s$ to $g(X(s),Y(s))$.

Independence of r.v.s

Definition

Independence of two r.v.s

Random variables $X$ and $Y$ are said to be independent if

$$P(X\le x,Y\le y)=P(X\le x)P(Y\le y)$$

for all $x,y\in R$ In the discrete case, this is equivalent to the condition

$$P(X=x,Y=y)=P(X=x)P(Y=y),$$

for all $x$, $y$ with $x$ in the support of $X$ and $y$ in the support of $Y$.

Definition

Independence of many r.v.s

Random variables $X_1,X_2,\dots ,X_n$ are independent if

$$P(X_1\le x_1,...,X_n\le x_n)=P(X_1\le x_1)...P(X_n\le x_n),$$

for all $x_1,...,x_n\in R$. For infinitely many r.v.s, we say that they are independent if every finite subset of the r.v.s is independent.

Theorem

Functions of independent r.v.s

If $X$ and $Y$ are independent r.v.s, then any function of $X$ is independent of any function of $Y$.

Definition

i.i.d.

We will often work with random variables that are independent and have the same distribution. We call such r.v.s independent and identically distributed, or i.i.d. for short.

Theorem

If $X\sim Bin(n,p)$, viewed as the number of successes in $n$ independent Bernoulli trials with success probability $p$, then we can write $X=X_1+\cdots +X_n$ where the $X_i$ are i.i.d. $Bern(p)$.

Theorem

If $X\sim Bin(n,p)$, $Y\sim Bin(m,p)$, and $X$ is independent of $Y$ , then $X+Y\sim Bin(n+m,p)$.

Definition

Conditional independence of r.v.s

Random variables $X$ and $Y$ are conditionally independent given an r.v. $Z$ if for all $x,y\in R$ and all $z$ in the support of $Z$,

$$P(X\le x,Y\le y|Z=z)=P(X\le x|Z=z)P(Y\le y|Z=z).$$

For discrete r.v.s, an equivalent definition is to require

$$P(X=x,Y=y|Z=z)=P(X=x|Z=z)P(Y=y|Z=z).$$

Definition

Conditional PMF

For any discrete r.v.s $X$ and $Z$, the function $P(X=x|Z=z)$, when considered as a function of $x$ for fixed $z$, is called the conditional PMF of $X$ given $Z=z$.

Connections between Binomial and Hypergeometric

Theorem

If $X\sim Bin(n,p)$, $Y\sim Bin(m,p)$, and $X$ is independent of $Y$ , then the conditional distribution of $X$ given $X+Y=r$ is $HGeom(n,m,r)$.

Theorem

If $X\sim HGeom(w,b,n)$ and $N=w+b\to \infty$ such that $p=w/(w+b)$ remains fixed, then the PMF of $X$ converges to the $Bin(n,p)$ PMF.

Solutions to chapter exercises

Previous Chapter | Next Chapter