108 Statistical Test of the difference between Means -- Independent/Unpaired Samples
108.1 Theory
We define a first population \(X_1 \sim \text{N}\left( \mu_1, \sigma_1^2 \right)\) from which a simple random sample is drawn of size \(n_1\) with sample mean \(\bar{x}_1 \sim \text{N} \left( \mu_1, \frac{\sigma_1^2}{n_1} \right)\).
We also define a second population \(X_2 \sim \text{N}\left( \mu_2, \sigma_2^2 \right)\) from which a simple random sample is drawn of size \(n_2\) with sample mean \(\bar{x}_2 \sim \text{N} \left( \mu_2, \frac{\sigma_2^2}{n_2} \right)\).
To test the difference between the sample means one can use the following equation:
\[ \left( \bar{x}_1 - \bar{x}_2 \right) \sim \text{N} \left( \left( \mu_1 - \mu_2 \right), \frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2} \right) \]
Depending on \(\sigma_1\) and \(\sigma_2\) one can distinguish between four cases, each of which is discussed in turn.
108.1.1 Case 1: \(\sigma_1\) and \(\sigma_2\) are known and unequal
The test statistic is defined as
\[ u = \frac{\left( \bar{x}_1 - \bar{x}_2 \right) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2} }} \]
for which it can be shown that \(u \sim \text{N}(0,1)\).
108.1.2 Case 2: \(\sigma_1\) and \(\sigma_2\) are known and equal
The test statistic is defined as
\[ u = \frac{\left( \bar{x}_1 - \bar{x}_2 \right) - \left( \mu_1 - \mu_2 \right) }{\sqrt{\sigma^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right) }} = \frac{\left( \bar{x}_1 - \bar{x}_2 \right) - (\mu_1 - \mu_2)}{\sigma \sqrt{\frac{n_1 + n_2}{n_1 \times n_2}}} \]
for which it can be shown that \(u \sim \text{N}(0,1)\).
108.1.3 Case 3: \(\sigma_1\) and \(\sigma_2\) are unknown but equal
108.1.3.1 Unequal Sample Sizes
The test statistic is defined as
\[ t = \frac{\left( \bar{x}_1 - \bar{x}_2 \right) - (\mu_1 - \mu_2)}{\sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right) }} = \frac{\left( \bar{x}_1 - \bar{x}_2 \right) - (\mu_1 - \mu_2)}{s_p \sqrt{\frac{n_1 + n_2}{n_1 \times n_2}}} \]
where the pooled variance \(s_p\) is defined as
\[ s_p^2 = \frac{s_1^2(n_1- 1) + s_2^2 (n_2 -1)}{n_1 + n_2 - 2} \]
and where the sample variances are estimated by
\[ s_i^2 = \frac{\sum_{j=1}^{n_i} \left( x_{ij} - \bar{x}_i \right)^2}{n_i - 1} \]
for \(i = 1, 2\).
It can be shown that \(t \sim t_{n_1+n_2-2}\).
108.1.3.2 Equal Sample Sizes
As shown in the previous description, the denominator of the test statistic is
\[ \sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)} \]
where the pooled sample variance is
\[ s_p^2 = \frac{s_1^2 (n_1 - 1) + s_2^2(n_2 - 1)}{n_1 + n_2 - 2} \]
When both samples have the same size, the pooled variance becomes
\[ s_p^2 = \frac{(n-1)(s_1^2 + s_2^2)}{2(n-1)} = \frac{s_1^2 + s_2^2}{2} \]
Substituting this pooled variance into the denominator of the test statistic results in
\[ \sqrt{\frac{s_1^2 + s_2^2}{2} \left( \frac{1}{n} + \frac{1}{n} \right)} = \sqrt{\frac{s_1^2 + s_2^2}{2} \times \frac{2}{n}} = \sqrt{\frac{s_1^2 + s_2^2}{n}} \]
108.1.4 Case 4: \(\sigma_1\) and \(\sigma_2\) are unknown and unequal
Unequal Sample Sizes
The test statistic is defined as
\[ t = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}} \]
where the sample variances are estimated by
\[ s_i^2 = \frac{\sum_{j=1}^{n_i} \left( x_{ij} - \bar{x}_i \right)^2}{n_i - 1} \]
with \(i = 1, 2\).
It can be shown that \(t \sim t_{\Delta}\) with \(\Delta\) degrees of freedom which can be computed as follows (Welch 1947; Satterthwaite 1946)
\[ \Delta = \frac{\left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)^2}{\frac{\left( \frac{s_1^2}{n_1} \right)^2}{n_1-1} + \frac{\left( \frac{s_2^2}{n_2} \right)^2}{n_2-1}} \]
The degrees of freedom \(\Delta\) will not necessarily be a whole number. As a close approximation the next smaller integer can be used.
108.1.4.1 Equal Sample Sizes
If both samples have the same size, the third and fourth test statistic are equal. The denominator of the test statistic for the 4th case is defined as
\[ \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}} \]
When both samples have the same sample size this becomes
\[ \sqrt{\frac{s_1^2}{n} + \frac{s_2^2}{n}} = \sqrt{\frac{s_1^2 + s_2^2}{n}} \]
108.2 Examples
108.2.1 Case 1: \(\sigma_1\) and \(\sigma_2\) are known and unequal
108.2.1.1 Problem
| Expected Value (Population 1) | \(\mu_1\) | ? |
| Variance (Population 1) | \(\sigma_1^2\) | 25 |
| Expected Value (Population 2) | \(\mu_2\) | ? |
| Variance (Population 2) | \(\sigma_2^2\) | 9 |
| Size of Sample 1 | \(n_1\) | 15 |
| Mean of Sample 1 | \(\bar{x}_1\) | 100 |
| Size of Sample 2 | \(n_2\) | 10 |
| Mean of Sample 2 | \(\bar{x}_2\) | 103 |
| Test Value H\(_0\) | \(\mu_1 - \mu_2\) | 0 |
| Type I error | \(\alpha\) | 0.05 |
| Critical Value | \(c\) | ? |
We use the left-sided alternative \(H_A: \mu_1 - \mu_2 < 0\), so the critical region is of the form \(\bar{x}_1 - \bar{x}_2 \leq c\).
108.2.1.2 Critical Value (Region)
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq c \right) = \alpha = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{c - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{c - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) = 0.05 \]
\[ \text{P}(u \leq -1.644854) = 0.05 \]
Hence
\[ \begin{align*}\frac{c}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} &= -1.644854 \\c &= -1.644854 \times \sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}} \\&= -1.644854 \times \sqrt{\frac{25}{15} + \frac{9}{10}} \\&= -1.644854 \times \sqrt{2.56667} \\&= -2.635190\end{align*} \]
Since \(\bar{x}_1 - \bar{x}_2 = -3 < c = -2.63519\), it follows that we should reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.1.3 P-value
\[ \text{P}(\bar{x}_1 - \bar{x}_2 \leq -3) \]
\[ \begin{align*}\text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{-3 - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) &= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{-3 - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) \\&= \text{P} \left( u \leq \frac{-3}{\sqrt{\frac{25}{15} + \frac{9}{10}}} \right) \\&= \text{P} \left( u \leq \frac{-3}{\sqrt{2.566667}} \right) \\&= \text{P} (u \leq -1.872563) \\&= 1 - 0.969436 \\&= 0.030564\end{align*} \]
Since the p-value is smaller than \(\alpha = 0.05\) we reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.2 Case 2: \(\sigma_1\) and \(\sigma_2\) are known and equal
108.2.2.1 Problem
| Expected Value (Population 1) | \(\mu_1\) | ? |
| Variance (Population 1) | \(\sigma_1^2\) | 16 |
| Expected Value (Population 2) | \(\mu_2\) | ? |
| Variance (Population 2) | \(\sigma_2^2\) | 16 |
| Size of Sample 1 | \(n_1\) | 15 |
| Mean of Sample 1 | \(\bar{x}_1\) | 100 |
| Size of Sample 2 | \(n_2\) | 10 |
| Mean of Sample 2 | \(\bar{x}_2\) | 103 |
| Test Value H\(_0\) | \(\mu_1 - \mu_2\) | 0 |
| Type I error | \(\alpha\) | 0.05 |
| Critical Value | \(c\) | ? |
108.2.2.2 Critical Value (Region)
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq c \right) = \alpha = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{c - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{c - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) = 0.05 \]
\[ \text{P}(u \leq -1.644854) = 0.05 \]
Hence
\[ \begin{align*}\frac{c}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} &= -1.644854 \\c &= -1.644854 \times \sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}} \\&= -1.644854 \times \sqrt{\frac{16}{15} + \frac{16}{10}} \\&= -1.644854 \times \sqrt{2.6667} \\&= -2.686035\end{align*} \]
Since \(\bar{x}_1 - \bar{x}_2 = -3 < c = -2.686035\), it follows that we should reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.2.3 P-value
\[ \text{P}(\bar{x}_1 - \bar{x}_2 \leq -3) \]
\[ \begin{align*}\text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{-3 - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) &= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \leq \frac{-3 - 0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} \right) \\&= \text{P} \left( u \leq \frac{-3}{\sqrt{\frac{16}{15} + \frac{16}{10}}} \right) \\&= \text{P} \left( u \leq \frac{-3}{\sqrt{2.66667}} \right) \\&= \text{P} (u \leq -1.837117) \\&= 1 - 0.966904 \\&= 0.033096\end{align*} \]
Since the p-value is smaller than \(\alpha = 0.05\) we reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.3 Case 3: \(\sigma_1\) and \(\sigma_2\) are unknown but equal
108.2.3.1 Problem
| Expected Value (Population 1) | \(\mu_1\) | ? |
| Variance (Population 1) | \(\sigma_1^2\) | ? |
| Expected Value (Population 2) | \(\mu_2\) | ? |
| Variance (Population 2) | \(\sigma_2^2\) | ? |
| Size of Sample 1 | \(n_1\) | 15 |
| Mean of Sample 1 | \(\bar{x}_1\) | 100 |
| Variance of Sample 1 | \(s_1^2\) | 25 |
| Size of Sample 2 | \(n_2\) | 10 |
| Mean of Sample 2 | \(\bar{x}_2\) | 103 |
| Variance of Sample 2 | \(s_2^2\) | 9 |
| Test Value H\(_0\) | \(\mu_1 - \mu_2\) | 0 |
| Type I error | \(\alpha\) | 0.05 |
| Critical Value | \(c\) | ? |
108.2.3.2 Critical Value (Region)
The pooled variance is
\[ \begin{align*}s_p^2 &= \frac{s_1^2 (n_1-1) + s_2^2(n_2 -1)}{n_1 + n_2 - 2} \\&= \frac{25(15-1)+9(10-1)}{15+10-2}\\&= \frac{25 \times 14 + 9 \times 9}{23} \\&= \frac{350+81}{23} \\&= \frac{431}{23} \\&= 18.7391304\end{align*} \]
Hence
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq c \right) = \alpha = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \leq \frac{c-(\mu_1-\mu_2)}{\sqrt{s_p^2\left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \right) = 0.05 \]
\[ \text{P}\left( t_{23} \leq -1.713872 \right) = 0.05 \]
\[ \begin{align*}\frac{c}{\sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} &= -1.713872 \\c &= -1.713872 \times \sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)} \\&= -1.713872 \times \sqrt{18.73913 \left( \frac{1}{15} + \frac{1}{10} \right) } \\&= -1.713872 \times \sqrt{3.1231884} \\&= -3.028847 \end{align*} \]
Since \(\bar{x}_1 - \bar{x}_2 = -3 > c = -3.028847\) there is no reason to reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.3.3 P-value
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq -3 \right) = \text{p-value} \]
\[ \begin{align*}\text{p-value} &= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \leq \frac{-3 -(\mu_1-\mu_2)}{\sqrt{s_p^2\left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \right) \\&= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{s_p^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \leq \frac{-3 -0}{\sqrt{s_p^2\left( \frac{1}{n_1} + \frac{1}{n_2} \right)}} \right)\\&= \text{P} \left( t_{23} \leq \frac{-3}{\sqrt{18.7391304 \left( \frac{1}{15} + \frac{1}{10} \right)}} \right) \\&= \text{P} \left( t_{23} \leq \frac{-3}{\sqrt{18.7391304 \times \frac{25}{100}}} \right) \\&= \text{P} \left( t_{23} \leq \frac{-3}{\sqrt{3.1231884}} \right) \\&= \text{P} \left( t_{23} \leq -1.69754839 \right) \\&= 1-0.948457 \\&= 0.051543\end{align*} \]
Since the p-value (=0.051543) is larger than the chosen type I error (=\(\alpha = 0.05\)) there is no reason to reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\).
108.2.4 Case 4: \(\sigma_1\) and \(\sigma_2\) are unknown and unequal
108.2.4.1 Problem
| Expected Value (Population 1) | \(\mu_1\) | ? |
| Variance (Population 1) | \(\sigma_1^2\) | ? |
| Expected Value (Population 2) | \(\mu_2\) | ? |
| Variance (Population 2) | \(\sigma_2^2\) | ? |
| Size of Sample 1 | \(n_1\) | 15 |
| Mean of Sample 1 | \(\bar{x}_1\) | 100 |
| Variance of Sample 1 | \(s_1^2\) | 25 |
| Size of Sample 2 | \(n_2\) | 10 |
| Mean of Sample 2 | \(\bar{x}_2\) | 103 |
| Variance of Sample 2 | \(s_2^2\) | 9 |
| Test Value H\(_0\) | \(\mu_1 - \mu_2\) | 0 |
| Type I error | \(\alpha\) | 0.05 |
| Critical Value | \(c\) | ? |
108.2.4.2 Critical Value (Region)
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq c \right) = \alpha = 0.05 \]
\[ \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{ \left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \leq \frac{c-(\mu_1-\mu_2)}{\sqrt{\left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \right) = 0.05 \]
The number of degrees of freedom \(\Delta\) is given by
\[ \Delta = \frac{\left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)^2}{ \frac{\left( \frac{s_1^2}{n_1} \right)^2}{n_1-1} + \frac{\left( \frac{s_2^2}{n_2} \right)^2}{n_2-1}} = \frac{\left( \frac{250+135}{150} \right)^2}{\frac{\left( \frac{25}{15} \right)^2}{15-1} + \frac{\left( \frac{9}{10} \right)^2}{10-1} } = \frac{6.58778}{\frac{2.77778}{14} + \frac{0.81}{9}} = \frac{6.58778}{0.288413} = 22.8415 \]
It follows that
\[ \text{P}\left( t_{22} \leq -1.717144 \right) = 0.05 \]
\[ \begin{align*}\frac{c - (\mu_1 - \mu_2)}{\sqrt{ \left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} &= -1.717144 \\c &= -1.717144 \times \sqrt{ \left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)} \\&= -1.717144 \times \sqrt{ \frac{25}{15} + \frac{9}{10} } \\&= -1.717144 \times \sqrt{2.566667} \\&= -2.751006 \end{align*} \]
Since \(\bar{x}_1 - \bar{x}_2 = -3 < c = -2.751006\) we reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\) and accept the Alternative Hypothesis.
108.2.4.3 P-value
\[ \text{P}\left( \bar{x}_1 - \bar{x}_2 \leq -3 \right) = \text{p-value} \]
\[ \begin{align*}\text{p-value} &= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{ \left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \leq \frac{-3 -(\mu_1-\mu_2)}{\sqrt{\left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \right) \\&= \text{P} \left( \frac{(\bar{x}_1 - \bar{x}_2) - 0}{\sqrt{ \left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \leq \frac{-3 -0}{\sqrt{\left( \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} \right)}} \right)\\&= \text{P} \left( t_{22} \leq \frac{-3}{\sqrt{ \left( \frac{25}{15} + \frac{9}{10} \right)}} \right) \\&= \text{P} \left( t_{22} \leq \frac{-3}{\sqrt{ 2.566667}} \right) \\&= \text{P} \left( t_{22} \leq \frac{-3}{1.602082} \right) \\&= \text{P} \left( t_{22} \leq -1.872563 \right) \\&= 1-0.96276 \\&= 0.03724\end{align*} \]
Since the p-value (=0.03724) is smaller than the chosen type I error (=\(\alpha = 0.05\)) we reject the Null Hypothesis H\(_0: \mu_1 - \mu_2 = 0\) and accept the Alternative Hypothesis.