The Unpaired Two Sample t-Test is sometimes called “Independent Samples t-Test” and represents the default test used when the means of two independent samples are compared. When valid pairing is available with positive within-pair correlation, the paired test is usually more powerful (smaller type II error \(\beta\)) than the unpaired test (cfr. Chapter 116).
Note: when performing the Unpaired Two Sample t-Test, it is not required that the samples sizes are the same even though a highly unbalanced design (i.e. \(|n_1 - n_2|\) is large) might lead to problems when we wish to assess whether or not the underlying assumptions are satisfied.
118.1 Hypotheses – Examples
Suppose we wish to test the following two-sided statistical hypothesis for a bivariate, quantitative dataset:
The underlying theory is described in Chapter 108 (Statistical Test of the difference between Means – Independent/Unpaired Samples). The chosen type I error \(\alpha\) is 5%.
As explained in Chapter 108 there are four cases that can be considered:
Case 1: \(\sigma_1\) and \(\sigma_2\) are known and unequal
Case 2: \(\sigma_1\) and \(\sigma_2\) are known and equal
Case 3: \(\sigma_1\) and \(\sigma_2\) are unknown but equal
Case 4: \(\sigma_1\) and \(\sigma_2\) are unknown and unequal
In practice, however, the first two cases are not commonly encountered. This means that we only need to concentrate on cases 3 & 4. The Unpaired Two Sample t-Test corresponds to case 3 whereas the Welch Test is used when the variances are unequal (i.e. case 4).
ImportantDecision Threshold Choice
Main role (mean comparison): usually confirmatory. Choose and justify the significance level for the mean-difference claim (often 1% to 5% in confirmatory work).
Supporting role (variance ratio / F-test):diagnostic, not a strict gatekeeper. If used as an assumption check, a higher diagnostic alpha (e.g. 10% to 20%) may be reasonable to reduce false reassurance.
Interpretation caution: a non-significant F-test does not prove equal variances.
Robust alternative: Welch’s test (Chapter 119) is often a safer default and reduces dependence on a fragile gatekeeping diagnostic.
Reporting: for the mean test, report p-value, confidence interval, and an effect size (e.g. Cohen’s \(d\) or Hedges’ \(g\)).
For the general role-based framework, see Chapter 112.
Data X: a multivariate dataset containing quantitative data
Names of X columns: a space delimited list of names (one name for each column)
Column number of first sample: a positive integer value of the column in the multivariate dataset which corresponds to the first sample
Column number of second sample: a positive integer value of the column in the multivariate dataset which corresponds to the second sample
Confidence: this is \(1 - \alpha\) (i.e. 1 minus the chosen type I error)
Alternative: parameter which defines the type of Hypothesis Test to be computed. This parameter can be set to the following values:
two.sided
less
greater
Are observations paired?: This parameter can be set to the following values:
unpaired
paired
Null Hypothesis: this is the value of \(\mu_0\) against which the hypothesis is tested (often it is the case that \(\mu_0 = 0\))
118.2.3 Output
The following analysis shows the Unpaired Two Sample t-Test, comparing glucose levels of patients with and without diabetes type II, assuming equal variances:
The p-value for the two-sided hypothesis is smaller than the chosen type I error, i.e. \(p = 7.075e-13 < 0.05\). As a consequence we reject the Null Hypothesis.
The same conclusion can be drawn from the two-sided confidence interval, i.e. \(\mu_0 = 0 \notin [-40.15520,-23.75033]\).
The classical Unpaired Two Sample t-Test assumes that the Variances of both populations are equal. Therefore, it is useful to inspect the F-test in the lower part of the output table, but this should not be treated as a strict decision gate for selecting the mean test.
The assumption of equal Variances cannot be tested directly (we are treating case 3 where \(\sigma_1\) and \(\sigma_2\) are unknown). Hence, we have to rely on the sample Variances \(s_1^2\) and \(s_2^2\) instead. The Hypothesis Test for the Variances can be formulated as follows:
which can be solved with an F-Test as described in Chapter 110).
From the statistical output it can be concluded that the Null Hypothesis \(\text{H}_0: \sigma_1^2 = \sigma_2^2\) cannot be rejected because the null value (=1) is contained in the interval [0.5069535, 1.1724351]. Alternatively, one can use the p-value and compare it with a chosen type I error. In this case the p-value = 0.2336, so we fail to reject the Null Hypothesis of equal variances. This does not prove that the population variances are equal; it only indicates that this sample does not provide strong evidence against equality.
The conclusion is that: (a) this sample does not show strong evidence of unequal variances, so the classical Unpaired Two Sample t-Test may be reasonable, while Welch’s test remains a robust default; and (b) the means of both samples are significantly different from each other.
If we specify Alternative = "greater", we are testing \(\text{H}_A: \mu_1 > \mu_2\).
If we specify Alternative = "less", we are testing \(\text{H}_A: \mu_1 < \mu_2\). In this example, the p-value for less is 3.538e-13, so we reject the Null Hypothesis in that pre-specified direction.
The one-sided direction (less or greater) must be chosen a priori based on substantive theory, not selected after inspecting sample means.
The right-sided confidence interval allows us to draw the same conclusion because \(\mu_0 = 0 \notin [-\infty,-25.07898]\), which implies that we should reject the Null Hypothesis.
For reporting, include an effect size for mean differences (e.g. Cohen’s \(d\)(Cohen 2013) or Hedges’ \(g\)(Hedges 1981)), not only p-values:
\[
d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}.
\]
Note: it is common to inspect whether the Variances of both populations appear equal using the F-Test (as in the two-sided case), but a non-significant result does not establish equality.
To compute the Unpaired Two Sample t-Test on your local machine, the following script can be used in the R console.
Note: this local script is a synthetic template. The embedded app example above uses the Pima.tr dataset and therefore has different numeric output.
set.seed(123)A <-runif(15, 1, 7)B <-runif(15, 1, 7)x <-cbind(A, B)par1 =1#column number of first samplepar2 =2#column number of second samplepar3 =0.95#confidence (= 1 - alpha)par4 ='two.sided'par5 ='unpaired'par6 =0.0#Null Hypothesisif (par5 =='unpaired') paired <-FALSEelse paired <-TRUE(t.test(x[,par1], x[,par2], var.equal=TRUE, alternative=par4, paired=paired, mu=par6, conf.level=par3))(var.test(x[,par1], x[,par2], conf.level=par3))
Two Sample t-test
data: x[, par1] and x[, par2]
t = -0.049545, df = 28, p-value = 0.9608
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.362858 1.298488
sample estimates:
mean of x mean of y
4.418309 4.450493
F test to compare two variances
data: x[, par1] and x[, par2]
F = 0.91398, num df = 14, denom df = 14, p-value = 0.8687
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
0.3068516 2.7223826
sample estimates:
ratio of variances
0.9139844
In most datasets, unpaired groups are ordered in a long format. In this sense the previous R code is not representative because we created a dataset in wide format. Therefore the R module shown below uses two different variables (i.e. X and Y): the first variable (X) is a quantitative variable and the second (Y) contains the grouping (for a two sided test this grouping must be binary). Here is an example of how the test works in long format (note: observe how the t.test and `var.test” functions use “formulas” in the case of long format data):
x =data.frame(measurement =c(A, B), group =c(rep("A", 15), rep("B", 15)))par3 =0.95#confidence (= 1 - alpha)par4 ='two.sided'# par5 = 'unpaired'par6 =0.0# Null Hypothesis# if (par5 == 'unpaired') paired <- FALSE else paired <- TRUE(t.test(measurement ~ group, var.equal=TRUE, alternative=par4, mu=par6, conf.level=par3, data = x))(var.test(measurement ~ group, conf.level=par3, data = x))
Two Sample t-test
data: measurement by group
t = -0.049545, df = 28, p-value = 0.9608
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
-1.362858 1.298488
sample estimates:
mean in group A mean in group B
4.418309 4.450493
F test to compare two variances
data: measurement by group
F = 0.91398, num df = 14, denom df = 14, p-value = 0.8687
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
0.3068516 2.7223826
sample estimates:
ratio of variances
0.9139844
Note: in recent R versions, the argument paired must not be used in the t.test function when a formula is used because it is implicitly assumed to be unpaired.
118.3 Assumptions
The data from both samples are assumed to be independently and identically distributed. Similarly to the One Sample t-Test we either assume normality of the observations in the Populations or rely on the Central Limit Theorem to state that the sample means are normally distributed.
In addition, the Variances of both Populations are assumed to be equal (i.e. \(\sigma_1^2 = \sigma_2^2\)). In practice, these Variances are unknown which implies that the researcher must test this assumption based on the Sample Variances.
118.4 Alternatives
There are several alternatives for the Unpaired Two Sample t-Test:
The Wilcoxon rank-sum test (also known as the Mann-Whitney U Test)
Notched Boxplots
Bayesian tests
The Bootstrap Plot for differences in Central Tendency
Cohen, Jacob. 2013. Statistical Power Analysis for the Behavioral Sciences. Academic press.
Hedges, Larry V. 1981. “Distribution Theory for Glass’s Estimator of Effect Size and Related Estimators.”Journal of Educational Statistics 6 (2): 107–28. https://doi.org/10.2307/1164588.