Hypothesis Testing

In null hypothesis significant testing, we are testing the validity of a claim about a population against a counter claim using sample data.

$\large H_0$ : A hypothesis claiming no difference between sample and population

$\large H_A$ : $\large H_0$ , which is usually what we're trying to prove.

	$\large H_0$	$\large H_0$
$\large H_0$ is True	Correct Decision (True Negative)	Type I Error (False Positive)
$\large H_0$ is False	Type II Error (False Negative)	Correct Decision (True Positive)

$\large T$ : A statistic used to test our hypothesis.

$\large \alpha$ : $\large H_0$ . This is the probability of getting a type I error.

$\large p$ -value: $\large H_0$ is true.

Power of the Test: $\large H_0$ $\large H_A$ is true.

\begin{matrix} \begin{aligned} α & = p (reject H_{0} | H_{0}) \\ = p (Type I Error) \\ = p (False Positive) \end{aligned} \end{matrix}

\begin{matrix} \begin{aligned} 1 - α & = p (do not reject H_{0} | H_{A}) \\ = p (True Negative) \end{aligned} \end{matrix}

\begin{matrix} \begin{aligned} β & = p (do not reject H_{0} | H_{A}) \\ = p (Type II Error) \\ = p (False Negative) \end{aligned} \end{matrix}

\begin{matrix} \begin{aligned} 1 - β & = p (reject H_{0} | H_{A}) \\ = p (True Positive) \\ = Power of the Test \end{aligned} \end{matrix}

$z$ -Test

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$
$\Large x_i \sim N(\mu, \sigma^2)$
$\Large \mu$ $\Large \sigma^2$ is known

Null Hypothesis

H_{0} : μ = μ_{0}

Alternative Hypothesis

H_{A} : μ > μ_{0}, μ < μ_{0}, or μ \neq μ_{0}

Test Statistic

z = \frac{\bar{x} - μ_{0}}{σ / \sqrt{n}}

Null Distribution

\begin{matrix} Standard Normal Distribution: \\ Z \sim N (0, 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{Z \le z\}$	$\large \texttt{scipy.stats.norm.cdf(z, 0, 1)}$	$\large \texttt{pnorm(z, 0, 1)}$
Right-tailed	$\Large P\{Z \ge z\}$	$\large \texttt{1 - scipy.stats.norm.cdf(z, 0, 1)}$	$\large \texttt{1 - pnorm(z, 0, 1)}$
Two-tailed	$\Large P\{\abs{Z} \ge \abs{z}\}$	$\large \texttt{2 * (1 - scipy.stats.norm.cdf(abs(z), 0, 1))}$	$\large \texttt{2 * (1 - pnorm(abs(z), 0, 1))}$

$z$ -Test: Two Samples

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large y_1, y_2, ..., y_m$
$\Large x_i \sim N(\mu_x, \sigma_x^2)$ $\Large y_i \sim N(\mu_y, \sigma_y^2)$
$\Large \mu_x, \mu_y$ $\Large \sigma_x^2, \sigma_y^2$ are known

Null Hypothesis

H_{0} : μ_{x} - μ_{y} = μ_{0}

Alternative Hypothesis

H_{A} : μ_{x} - μ_{y} > μ_{0}, μ_{x} - μ_{y} < μ_{0}, or μ_{x} - μ_{y} \neq μ_{0}

Test Statistic

z = \frac{\bar{x} - \bar{y} - μ_{0}}{σ_{p}}

$\large \sigma_p^2$ is the pooled population variance:

σ_{p}^{2} = \frac{σ_{x}^{2}}{n} + \frac{σ_{y}^{2}}{m}

Null Distribution

\begin{matrix} Standard Normal Distribution: \\ Z \sim N (0, 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{Z \le z\}$	$\large \texttt{scipy.stats.norm.cdf(z, 0, 1)}$	$\large \texttt{pnorm(z, 0, 1)}$
Right-tailed	$\Large P\{Z \ge z\}$	$\large \texttt{1 - scipy.stats.norm.cdf(z, 0, 1)}$	$\large \texttt{1 - pnorm(z, 0, 1)}$
Two-tailed	$\Large P\{\abs{Z} \ge \abs{z}\}$	$\large \texttt{2 * (1 - scipy.stats.norm.cdf(abs(z), 0, 1))}$	$\large \texttt{2 * (1 - pnorm(abs(z), 0, 1))}$

$t$ -Test

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$
$\Large x_i \sim N(\mu, \sigma^2)$
$\Large \mu, \sigma^2$ are unknown

Null Hypothesis

H_{0} : μ = μ_{0}

Alternative Hypothesis

H_{A} : μ > μ_{0}, μ < μ_{0}, or μ \neq μ_{0}

Test Statistic

t = \frac{\bar{x} - μ_{0}}{s / \sqrt{n}}

Null Distribution

\begin{matrix} t-Distribution : \\ T \sim t (n - 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{T \le t\}$	$\large \texttt{scipy.stats.t.cdf(t, n - 1)}$	$\large \texttt{pt(t, n - 1)}$
Right-tailed	$\Large P\{T \ge t\}$	$\large \texttt{1 - scipy.stats.t.cdf(t, n - 1)}$	$\large \texttt{1 - pt(t, n - 1)}$
Two-tailed	$\Large P\{\abs{T} \ge \abs{t}\}$	$\large \texttt{2 * (1 - scipy.stats.t.cdf(abs(t), n - 1))}$	$\large \texttt{2 * (1 - pt(abs(t), n - 1))}$

$t$ -Test: Two Samples with Equal Variances

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large y_1, y_2, ..., y_m$
$\Large x_i \sim N(\mu_x, \sigma^2)$ $\Large y_i \sim N(\mu_y, \sigma^2)$
$\Large \mu_x, \mu_y, \sigma^2$ are unknown

Null Hypothesis

H_{0} : μ_{x} - μ_{y} = μ_{0}

Alternative Hypothesis

H_{A} : μ_{x} - μ_{y} > μ_{0}, μ_{x} - μ_{y} < μ_{0}, or μ_{x} - μ_{y} \neq μ_{0}

Test Statistic

t = \frac{\bar{x} - \bar{y} - μ_{0}}{s_{p} \sqrt{\frac{1}{n} + \frac{1}{m}}}

$\large s_p^2$ is the pooled sample variance:

s_{p}^{2} = \frac{(n - 1) s_{x}^{2} + (m - 1) s_{y}^{2}}{n + m - 2}

Null Distribution

\begin{matrix} t-Distribution : \\ T \sim t (n + m - 2) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{T \le t\}$	$\large \texttt{scipy.stats.t.cdf(t, n + m - 2)}$	$\large \texttt{pt(t, n + m - 2)}$
Right-tailed	$\Large P\{T \ge t\}$	$\large \texttt{1 - scipy.stats.t.cdf(t, n + m - 2)}$	$\large \texttt{1 - pt(t, n + m - 2)}$
Two-tailed	$\Large P\{\abs{T} \ge \abs{t}\}$	$\large \texttt{2 * (1 - scipy.stats.t.cdf(abs(t), n + m - 2))}$	$\large \texttt{2 * (1 - pt(abs(t), n + m - 2))}$

$t$ -Test: Two Samples with Unequal Variances

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large y_1, y_2, ..., y_m$
$\Large x_i \sim N(\mu_x, \sigma_x^2)$ $\Large y_i \sim N(\mu_y, \sigma_y^2)$
$\Large \mu_x, \mu_y, \sigma_x^2, \sigma_y^2$ are unknown

Null Hypothesis

H_{0} : μ_{x} - μ_{y} = μ_{0}

Alternative Hypothesis

H_{A} : μ_{x} - μ_{y} > μ_{0}, μ_{x} - μ_{y} < μ_{0}, or μ_{x} - μ_{y} \neq μ_{0}

Test Statistic

t = \frac{\bar{x} - \bar{y} - μ_{0}}{s_{p} \sqrt{\frac{1}{n} + \frac{1}{m}}}

$\large s_p^2$ is the pooled sample variance:

s_{p}^{2} = s_{x}^{2} + s_{y}^{2}

Null Distribution

\begin{matrix} t -Distribution : \\ T \sim t (d f) \end{matrix}

$\large df$ is the degrees of freedom, defined by,

d f = ⌊ \frac{(s_{x}^{2} / n + s_{y}^{2} / m)^{2}}{(s_{x}^{2} / n)^{2} / (n - 1) + (s_{y}^{2} / m)^{2} / (m - 1)} ⌉

$\large \lfloor a \rceil$ $\large a$ .

Test Type	p-value	Python	R
Left-tailed	$\Large P\{T \le t\}$	$\large \texttt{scipy.stats.t.cdf(t, df)}$	$\large \texttt{pt(t, df)}$
Right-tailed	$\Large P\{T \ge t\}$	$\large \texttt{1 - scipy.stats.t.cdf(t, df)}$	$\large \texttt{1 - pt(t, df)}$
Two-tailed	$\Large P\{\abs{T} \ge \abs{t}\}$	$\large \texttt{2 * (1 - scipy.stats.t.cdf(abs(t), df))}$	$\large \texttt{2 * (1 - pt(abs(t), df))}$

$F$ -Test for Equal Means (One Way ANOVA)

Setup & Assumptions

$\large n$ $\large m$ $\Large x_{ij}; \: i = 1, 2, ..., n; \: j = 1, 2, ..., m$
$\Large x_{ij} \sim N(\mu_i, \sigma^2)$
$\Large \mu_i, \sigma^2$ are unknown

Null Hypothesis

H_{0} : μ_{1} = μ_{2} = . . . = μ_{n}

Alternative Hypothesis

H_{A} : μ_{i} \neq μ_{j} for some i, j

Test Statistic

v = \frac{M S_{b}}{M S_{w}}

where,

\begin{matrix} Mean Squared Sum Within Sets, M S_{w} \\ = \frac{S S_{w}}{n (m - 1)} \\ Mean Squared Sum Between Sets, M S_{b} \\ = \frac{S S_{b}}{n - 1} \\ Squared Sum Within Sets, S S_{w} \\ = \sum_{i = 1}^{n} \sum_{j = 1}^{m} (x_{i j} - \bar{x_{i}})^{2} = \sum_{i = 1}^{n} (m - 1) s_{i}^{2} \\ Squared Sum Between Sets, S S_{b} \\ = \sum_{i = 1}^{n} m (\bar{x_{i}} - \bar{x})^{2} \\ Squared Sum Total, S S_{t} \\ = \sum_{i = 1}^{n} \sum_{j = 1}^{m} (x_{i j} - \bar{x})^{2} \\ S S_{t} = S S_{w} + S S_{b} \end{matrix}

Null Distribution

\begin{matrix} F -Distribution : \\ V \sim F (n - 1, n (m - 1)) \end{matrix}

Test Type	p-value	Python	R
Right-tailed	$\Large P\{V \ge v\}$	$\large \texttt{1 - scipy.stats.f.cdf(v, n - 1, n * (m - 1))}$	$\large \texttt{1 - pf(v, n - 1, n * (m - 1))}$

$F$ -Test for Equal Variances

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large y_1, y_2, ..., y_m$
$\Large x_i \sim N(\mu_x, \sigma_x^2)$ $\Large y_i \sim N(\mu_y, \sigma_y^2)$
$\Large \mu_x, \mu_y, \sigma_x^2, \sigma_y^2$ are unknown

Null Hypothesis

H_{0} : σ_{x} = σ_{y}

Alternative Hypothesis

H_{A} : σ_{x} > σ_{y}, σ_{x} < σ_{y} or σ_{x} \neq σ_{y}

Test Statistic

v = \frac{s_{x}^{2}}{s_{y}^{2}}

Null Distribution

\begin{matrix} F -Distribution : \\ V \sim F (n - 1, m - 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{V \le v\}$	$\large \texttt{scipy.stats.f.cdf(v, n - 1, m - 1)}$	$\large \texttt{pf(v, n - 1, m - 1)}$
Right-tailed	$\Large P\{V \ge v\}$	$\large \texttt{1 - scipy.stats.f.cdf(v, n - 1, m - 1)}$	$\large \texttt{1 - pf(v, n - 1, m - 1)}$
Two-tailed	$\Large P\{\abs{V} \ge \abs{v}\}$	$\large \texttt{2 * min(1 - scipy.stats.f.cdf(v, n - 1, m - 1), scipy.stats.f.cdf(v, n - 1, m - 1))}$	$\large \texttt{2 * min(1 - pf(v, n - 1, m - 1), pf(v, n - 1, m - 1))}$

$\chi^2$ -Test for Variance

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$
$\Large x_i \sim N(\mu, \sigma^2)$
$\Large \mu, \sigma^2$ are unknown

Null Hypothesis

H_{0} : σ = σ_{0}

Alternative Hypothesis

H_{A} : σ > σ_{0}, σ < σ_{0} or σ \neq σ_{0}

Test Statistic

u = \frac{(n - 1) s^{2}}{σ_{0}^{2}}

Null Distribution

\begin{matrix} χ^{2} -Distribution : \\ U \sim χ^{2} (n - 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Left-tailed	$\Large P\{U \le u\}$	$\large \texttt{scipy.stats.chi2.cdf(u, n - 1)}$	$\large \texttt{pchisq(u, n - 1)}$
Right-tailed	$\Large P\{U \ge u\}$	$\large \texttt{1 - scipy.stats.chi2.cdf(u, n - 1)}$	$\large \texttt{1 - pchisq(u, n - 1)}$
Two-tailed	$\Large P\{\abs{U} \ge \abs{u}\}$	$\large \texttt{2 * min(1 - scipy.stats.chi2.cdf(u, n - 1), scipy.stats.chi2.cdf(u, n - 1))}$	$\large \texttt{2 * min(1 - pchisq(u, n - 1), pchisq(u, n - 1))}$

$\chi^2$ -Test for Goodness of Fit: Single Specific Distribution

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large x_i \in \{\omega_1, \omega_2, ..., \omega_K\}$
$\Large \{p_i\}^K_{i = 1}$
$\Large O_1, O_2, ..., O_K$ $\Large O_i$ $\Large \omega_i$ $\Large x_1, x_2, ..., x_n$ $\Large O_1 + O_2 + ... + O_K = n$

Null Hypothesis

H_{0} : x_{i} follows distribution {p_{i}}_{i = 1}^{K}

Alternative Hypothesis

H_{A} : x_{i} does not follow distribution {p_{i}}_{i = 1}^{K}

Test Statistic

u = \sum_{i = 1}^{K} \frac{(O_{i} - n p_{i})^{2}}{n p_{i}}

Null Distribution

\begin{matrix} χ^{2} -Distribution : \\ U \sim χ^{2} (K - 1) \end{matrix}

p-Value

Test Type	p-value	Python	R
Right-tailed	$\Large P\{U \ge u\}$	$\large \texttt{1 - scipy.stats.chi2.cdf(u, K - 1)}$	$\large \texttt{1 - pchisq(u, K - 1)}$

$\chi^2$ -Test for Goodness of Fit: Statistical Model

Setup & Assumptions

$\Large x_1, x_2, ..., x_n$ $\Large x_i \in \{\omega_1, \omega_2, ..., \omega_K\}$
$\Large \{(p_1(\theta), p_2(\theta), ..., p_K(\theta)) : \theta \in \Theta\}$
$\Large O_1, O_2, ..., O_K$ $\Large O_i$ $\Large \omega_k$ $\Large x_1, x_2, ..., x_n$ $\Large O_1 + O_2 + ... + O_K = n$

Null Hypothesis

H_{0} : x_{i} follows model {(p_{1} (θ), p_{2} (θ), . . ., p_{K} (θ)) : θ \in Θ}

Alternative Hypothesis

H_{A} : x_{i} does not follow model {(p_{1} (θ), p_{2} (θ), . . ., p_{K} (θ)) : θ \in Θ}

Test Statistic

u = \sum_{k = 1}^{K} \frac{(O_{k} - n p_{k} (\hat{θ}))^{2}}{n p_{k} (\hat{θ})}

$\large \hat{\theta}$ $\large \theta$ $\large H_0$ .

Null Distribution

\begin{matrix} χ^{2} -Distribution : \\ U \sim χ^{2} (K - 1 - d i m (Θ)) \end{matrix}

$\large dim(\Theta)$ $\large \Theta$ .

p-Value

Test Type	p-value	Python	R
Right-tailed	$\Large P\{U \ge u\}$	$\large \texttt{1 - scipy.stats.chi2.cdf(u, K - 1 - dim)}$	$\large \texttt{1 - pchisq(u, K - 1 - dim)}$

$\chi^2$ -Test for Goodness of Fit: Independence

Setup & Assumptions

$\Large (x_1, y_1), (x_2, y_2), ..., (x_n, y_n)$ $\Large x_i \in \{\omega_1, \omega_2, ..., \omega_K\}$ $\Large y_i \in \{\psi_1, \psi_2, ..., \psi_L\}$
$\Large p_{k,l} = P\{X =\omega_k,Y = \psi_l\}$
$\Large O_{i,.}$ $\Large \omega_k$ $\large x_i$ $\Large O_{.,j}$ $\Large \psi_k$ $\large y_i$ $\Large \sum\limits^{K}_{k = 1} \sum\limits^{L}_{l = 1} O_{k, l} = n$
$\Large \hat{p}_{k,.} = \frac{O_{k,.}}{n}, \hat{p}_{.,l} = \frac{O_{.,l}}{n}$

Null Hypothesis

\begin{matrix} H_{0} : X & Y are independent \\ p_{k, l} = p_{k, .} \times p_{., l} \end{matrix}

Alternative Hypothesis

\begin{matrix} H_{0} : X & Y are NOT independent \\ p_{k, l} \neq p_{k, .} \times p_{., l} for some k, l \end{matrix}

Test Statistic

u = \sum_{k = 1}^{K} \sum_{l = 1}^{L} \frac{(O_{k, l} - n {\hat{p}}_{k, .} {\hat{p}}_{., l})^{2}}{n {\hat{p}}_{k, .} {\hat{p}}_{., l}}

Null Distribution

\begin{matrix} χ^{2} -Distribution : \\ U \sim χ^{2} ((K - 1) (L - 1)) \end{matrix}

p-Value

Test Type	p-value	Python	R
Right-tailed	$\Large P\{U \ge u\}$	$\large \texttt{1 - scipy.stats.chi2.cdf(u, (K - 1) * (L - 1)}$	$\large \texttt{1 - pchisq(u, (K - 1) * (L - 1))}$

Hypothesis Testing

zz-Test

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

zz-Test: Two Samples

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

tt-Test

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

tt-Test: Two Samples with Equal Variances

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

tt-Test: Two Samples with Unequal Variances

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

FF-Test for Equal Means (One Way ANOVA)

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

FF-Test for Equal Variances

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

χ2\chi^2-Test for Variance

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

χ2\chi^2-Test for Goodness of Fit: Single Specific Distribution

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

χ2\chi^2-Test for Goodness of Fit: Statistical Model

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

χ2\chi^2-Test for Goodness of Fit: Independence

Setup & Assumptions

Null Hypothesis

Alternative Hypothesis

Test Statistic

Null Distribution

p-Value

$z$ -Test

$z$ -Test: Two Samples

$t$ -Test

$t$ -Test: Two Samples with Equal Variances

$t$ -Test: Two Samples with Unequal Variances

$F$ -Test for Equal Means (One Way ANOVA)

$F$ -Test for Equal Variances

$\chi^2$ -Test for Variance

$\chi^2$ -Test for Goodness of Fit: Single Specific Distribution

$\chi^2$ -Test for Goodness of Fit: Statistical Model

$\chi^2$ -Test for Goodness of Fit: Independence